{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第四章 神经网络工具箱nn\n",
    "上一章中提到，使用autograd可实现深度学习模型，但其抽象程度较低，如果用其来实现深度学习模型，则需要编写的代码量极大。在这种情况下，torch.nn应运而生，其是专门为深度学习而设计的模块。torch.nn的核心数据结构是`Module`，它是一个抽象概念，既可以表示神经网络中的某个层（layer），也可以表示一个包含很多层的神经网络。在实际使用中，最常见的做法是继承`nn.Module`，撰写自己的网络/层。下面先来看看如何用nn.Module实现自己的全连接层。全连接层，又名仿射层，输出$\\textbf{y}$和输入$\\textbf{x}$满足$\\textbf{y=Wx+b}$，$\\textbf{W}$和$\\textbf{b}$是可学习的参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch as t\n",
    "from torch import nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Linear(nn.Module): # 继承nn.Module\n",
    "    def __init__(self, in_features, out_features):\n",
    "        super(Linear, self).__init__() # 等价于nn.Module.__init__(self)\n",
    "        self.w = nn.Parameter(t.randn(in_features, out_features))\n",
    "        self.b = nn.Parameter(t.randn(out_features))\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = x.mm(self.w) # x.@(self.w)\n",
    "        return x + self.b.expand_as(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-2.4421, -0.2355,  0.5636],\n",
       "        [-1.4299, -1.0916, -1.9299]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer = Linear(4,3)\n",
    "input = t.randn(2,4)\n",
    "output = layer(input)\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w Parameter containing:\n",
      "tensor([[ 0.0481, -0.5937, -1.0651],\n",
      "        [ 0.5725,  0.1317, -0.0920],\n",
      "        [ 0.7409,  0.3163, -1.3940],\n",
      "        [ 0.1598, -0.3302,  0.3646]])\n",
      "b Parameter containing:\n",
      "tensor([-1.5881, -0.8368, -0.5220])\n"
     ]
    }
   ],
   "source": [
    "for name, parameter in layer.named_parameters():\n",
    "    print(name, parameter) # w and b "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见，全连接层的实现非常简单，其代码量不超过10行，但需注意以下几点：\n",
    "- 自定义层`Linear`必须继承`nn.Module`，并且在其构造函数中需调用`nn.Module`的构造函数，即`super(Linear, self).__init__()` 或`nn.Module.__init__(self)`，推荐使用第一种用法，尽管第二种写法更直观。\n",
    "- 在构造函数`__init__`中必须自己定义可学习的参数，并封装成`Parameter`，如在本例中我们把`w`和`b`封装成`parameter`。`parameter`是一种特殊的`Tensor`，但其默认需要求导（requires_grad = True），感兴趣的读者可以通过`nn.Parameter??`，查看`Parameter`类的源代码。\n",
    "- `forward`函数实现前向传播过程，其输入可以是一个或多个tensor。\n",
    "- 无需写反向传播函数，nn.Module能够利用autograd自动实现反向传播，这点比Function简单许多。\n",
    "- 使用时，直观上可将layer看成数学概念中的函数，调用layer(input)即可得到input对应的结果。它等价于`layers.__call__(input)`，在`__call__`函数中，主要调用的是 `layer.forward(x)`，另外还对钩子做了一些处理。所以在实际使用中应尽量使用`layer(x)`而不是使用`layer.forward(x)`，关于钩子技术将在下文讲解。\n",
    "- `Module`中的可学习参数可以通过`named_parameters()`或者`parameters()`返回迭代器，前者会给每个parameter都附上名字，使其更具有辨识度。\n",
    "\n",
    "可见利用Module实现的全连接层，比利用`Function`实现的更为简单，因其不再需要写反向传播函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Module能够自动检测到自己的`Parameter`，并将其作为学习参数。除了`parameter`之外，Module还包含子`Module`，主Module能够递归查找子`Module`中的`parameter`。下面再来看看稍微复杂一点的网络，多层感知机。\n",
    "\n",
    "多层感知机的网络结构如图4-1所示，它由两个全连接层组成，采用$sigmoid$函数作为激活函数，图中没有画出。\n",
    "![图4-1；多层感知机](imgs/multi_perceptron.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Perceptron(nn.Module):\n",
    "    def __init__(self, in_features, hidden_features, out_features):\n",
    "        nn.Module.__init__(self)\n",
    "        self.layer1 = Linear(in_features, hidden_features) # 此处的Linear是前面自定义的全连接层\n",
    "        self.layer2 = Linear(hidden_features, out_features)\n",
    "    def forward(self,x):\n",
    "        x = self.layer1(x)\n",
    "        x = t.sigmoid(x)\n",
    "        return self.layer2(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "layer1.w torch.Size([3, 4])\n",
      "layer1.b torch.Size([4])\n",
      "layer2.w torch.Size([4, 1])\n",
      "layer2.b torch.Size([1])\n"
     ]
    }
   ],
   "source": [
    "perceptron = Perceptron(3,4,1)\n",
    "for name, param in perceptron.named_parameters():\n",
    "    print(name, param.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见，即使是稍复杂的多层感知机，其实现依旧很简单。 构造函数`__init__`中，可利用前面自定义的Linear层(module)，作为当前module对象的一个子module，它的可学习参数，也会成为当前module的可学习参数。\n",
    "\n",
    "module中parameter的命名规范：\n",
    "- 对于类似`self.param_name = nn.Parameter(t.randn(3, 4))`，命名为`param_name`\n",
    "- 对于子Module中的parameter，会其名字之前加上当前Module的名字。如对于`self.sub_module = SubModel()`，SubModel中有个parameter的名字叫做param_name，那么二者拼接而成的parameter name 就是`sub_module.param_name`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为方便用户使用，PyTorch实现了神经网络中绝大多数的layer，这些layer都继承于nn.Module，封装了可学习参数`parameter`，并实现了forward函数，且很多都专门针对GPU运算进行了CuDNN优化，其速度和性能都十分优异。本书不准备对nn.Module中的所有层进行详细介绍，具体内容读者可参照官方文档[^1]或在IPython/Jupyter中使用nn.layer?来查看。阅读文档时应主要关注以下几点：\n",
    "\n",
    "- 构造函数的参数，如nn.Linear(in_features, out_features, bias)，需关注这三个参数的作用。\n",
    "- 属性、可学习参数和子module。如nn.Linear中有`weight`和`bias`两个可学习参数，不包含子module。\n",
    "- 输入输出的形状，如nn.linear的输入形状是(N, input_features)，输出为(N，output_features)，N是batch_size。\n",
    "\n",
    "这些自定义layer对输入形状都有假设：输入的不是单个数据，而是一个batch。输入只有一个数据，则必须调用`tensor.unsqueeze(0)` 或 `tensor[None]`将数据伪装成batch_size=1的batch\n",
    "\n",
    "[^1]: http://pytorch.org/docs/nn.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面将从应用层面出发，对一些常用的layer做简单介绍，更详细的用法请查看文档，这里只作概览参考。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 常用神经网络层\n",
    "#### 4.1.1 图像相关层\n",
    "\n",
    "图像相关层主要包括卷积层（Conv）、池化层（Pool）等，这些层在实际使用中可分为一维(1D)、二维(2D)、三维（3D），池化方式又分为平均池化（AvgPool）、最大值池化（MaxPool）、自适应池化（AdaptiveAvgPool）等。而卷积层除了常用的前向卷积之外，还有逆卷积（TransposeConv）。下面举例说明一些基础的使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMgAAADICAAAAACIM/FCAABaCUlEQVR4nGX9+dMlWZIdhh13vzci\n3vJtuWfWXl29THfPTnAGhiFpBAmSv0lmNOnvk2SizARBIo0gODAAJAAS2wCD6Zneqpfaq3Jfvu1t\nEfe6H/1w42uMTFXWbVkvX+b3XkRcX46fc1z+b4Q6QysI0WzakbHfXr68KF0yMVdQAddsqoYUWYhE\nRyiQAiIAQBpDRQgKPCAQIagC8UhpcgNcBK4UEuIenHzsVkfDkLLGASIK0QqCsFANk8pbby9TwuGL\nf/vSug4JISpVQRWRCJiGsAT7949fKwAQIPDXfkWQ7T8J3vwHfvMWEIC0/5ebl29ex///2+e/6T/8\nDv/a/37zJs7vkr/+OQAAqn/tvQBF8Nd/hIB1TEGQDgkh1BUKIFgLyIpkTgBKAcIgAkcIEAAURABC\nyvxXkAChUBIQcQqBgEQVCQjIaB9Iov0hcS+dawgsEEpRYbtCDDAQu11vlHxy93UNigsEFIgI2s8j\nBIYsX5w/TlUgAKABhKi2T+UAQ8iqFEEAEISAImRA5D/cthABIRQhQkCGQoKCAOevhri5T4RDKUFo\nBcTqIWUzV1VQDAQMQSGUAEP2rwdT1eP3Xj33mlTIUKGAFEAgAkhO/vrTV1UFAapCBAJTIUmG0wkg\n2oNPBCEASaoACIoCArKCQZFo34sUiJAi858QFSCCYgKIqApGUFQAFRErh1I9ov2WJlWBiChIkkRc\nXhanDPffG9w9CAgDEYEAKBKgZXn56+dFVKAkBIRIyiZgeIVQRYQmqiqAMDhfWSJIantIBcIQBaEQ\nhqpAhRARaQ+JqoooQCoEJAUhEAUSwKScxogIWDJTtZRERNohCAIyvdk5oet3HygjIoJQiCAEKgKo\n2fj4kyuKKtGeA4CaVILe3h9gUAJkQNoTQpkvsghBiAg1iQhFlYQIBRRRUUFA2/lUEQjkJnpQlxDQ\nlBAzUR+niHAkE1FARNQAUMEAweuLKYh8+8Mj92ihQBRQAVQt5Tx++fHLHRhKiEBUINrCJ52E5qwQ\nDWE7QSKYQ14EICJAu3ZBRnsMCBAipLPFGWmfvT21FqCogqRIkBBVcUjUqQQBigEqUEBM20cSitRX\nm0rK4uEjiXAQAm1PqahatsOXn1yZKaAIQCAQAiKMYIQIIICAkhntngmCBCER83GIYDv0ZASChEQ4\nJeik0ANksN2MCECiBohgiDAoCYBoTI52oNrfrHQSaHEtgM2bQw3Y6QenQhJRwABpydTMDp9/uhGT\nUNP56EFUVSFkBCAa1YMgnKQoqHBSBC6miBa0IoI3QSwAUQToU7h7jSBB0quThLTDSYIS7aaSEoQo\n3NniJiJICKM9BO2GxpurQkp//0NzRwvQZLiHmOw/e3wASsBMoWCLzikJAA9AAK8eZY7IjDnACkXb\nPQgyKCoCMjxACsRMIiI85ntDtsshIsJ2d1r+NJIUUxFCNcIBEgSjvYkqIqIiIpDDm9EJO3r/bkS4\npnbIIyDYf/rLqygeFGPSFn2gkuzmxotSVLUFwzl9U4DQVnmIMCDzmTBSAIRoey0oIolAQKESyrl6\nkBbABaAoolIUmO+nQCUACFtGa185hJB4c9qbSnfno9cTWUxBCE1N9r/+dGuSYCLepRZEAbU5QLas\nrxBJYJjOFQp1zoqirUAIbXEA0p5sDSGgFSACQgEDMCGpjJv6AzpnUAABgiFwd4EAIUB7Simcb785\nds8GQdLVB29+5SIBhShEdfzq823SENMIbfkN0vJFy2gAI6IlT6FHUKL9RrSf0W6gMGJO9EHOtWB7\nMghQwxlsDz0EojIXTMKbjMoQiIUzIhiBdhnbAygkW+AWubp0iNjJ+0e1eDtRZunw+S+uwNRlBURr\ngsCkRcb27N98n6gwgiSEogBBQTCRAmr7oQFqtGexxTSKUERDEN4OA6gAJSiABBiqBIVBmjjpAUSE\ntOKtXY+5CJprBS0vT/pE7e5/cD6pBKlitv38sw3EoHARuqu12K4SXiNC2leSaDdFACAgDM7npd2f\nCAZFWr4QkgQEAYEoAZZaSCLoDIItw7ZbT7baj6SIaPt1eJAQIEA6naKtGiFJXr0ZnbSjDx4wangN\nlf0Xn27UxDpTDUaVRChFTd0DEiK8qaqkPecQtNdCqIAowFCJ+Tiy5e24KcYVrgiJEFKiVSot/yBM\nGSmEFIlAyx4iDEYIRL0VrAAp0WJjC+/wV2e9gfnWh69GA9T08PXnO9DM6A5G0BMogFqUSgBJNObQ\nB4gEdC55oRBtJWErOKRldrjonLuYEAxRUtOyo1efSq0a1u4a2pkDtH0vkQAixIPt47YLOPcbrqQz\nHAREtq9WJiqLd77+nBS1w+NPNhBTUyVDKy2SSECVpdYATFopI2I6F82iTkBETUIFc5UFaQ9Jaw6g\n7QorxXQA0nJtFkljvD7f7ScxFaWIQOe7Q8icVOceLQBo6By6QIG3Kg8gBP76tDNFPvvOm71qmr7+\nfAOltkjlAVASCKgxanGoyZxVWomIVsGzPdet6lORAEWp7Y2KokYKydT3SOpY9pROKV0Mx+9Mb15f\nHmqiRAKgflO9EaL1plEUkC4xh/NoPVOQLiSFwOHFKonp8u0PfqHZv/nsShGqBnHCXRRIDDGF1zqG\nJKAVhwRb9lYgKFQqaCqCCFOQKioEFRYUOgK0vuusloRhKa6SIg7FOhnefnvz+tkFVJ0a0rpnoLLl\nltBoBTklRLSFrQDAYLSoRSH9/Kg3RT799vl5PPvsGgKFIAoRSBKBRLEsXuo01pzmvksUgJNEKPwm\nMRtBaItNUGG7cKSBUMqyL2S4ZB9FtDA30IKWz87e+frLrdFAgQTnKspl7lhaVGknRFpean2PiN9E\nm93Tky4Rw8Nv/9XjzzZIKqoGMKp1EYAmiKqUMu33kZKqANpCKV0NrAoSkRHhEipsBYU42i8D0lmK\nw9hZ1XAMQj8kFXdxTYwkYKTVd+79+kkZWgVjcZNySReZQzd4UxDEb5CP1hA1wOHy2bpT6vr9X3x6\nTiEttYZXTYKhksSSllIOm11aa2sx5roIsLDWH0soENoKKhjbAQId+c64LIf9gXWXEDJEq3yM2Mug\nVU0pYPD20dkno0Os5e5WEUc4lQFI6x6DJhQp0lrOIERDQBWhv7ydRSUdl1eEaUML3GFamQBLahq1\nTturwwnQomvrv4Q+KZOEUiWgDCVAoc7PBsG82O9wMTpFQhkl+5gH8yyTiiZFb3Sq0AT9d+786rFr\nnsEezFk05hg4X3cXUVUndG5HfY53Yvunq5yFL35akwDJoIhCM3cHsqaUpdQ6Xl/GcasSMN9ddw1t\nBwJUaVUtAzegEhly7+p8ml8zRFBdorikQwxijN5ImDrEGLj1O8efbDwlZ6sFWpcSAcrcykDmGjkk\nRMG5cYNRlDx/eST+5h/+UimqSUI5hVp4CDJ3ydRLGXdXm67lutbOesNvFNB2C1wJhlChEtCGAuX8\npFYjVCqsqovDqqiOOaUsIuFqodEAJ/H+e7d//rgmzD11zGEDpioqTlICgqSt62kdAzHHR8SLO931\nP/4nbgIoQlGpWWrV1PFqlzRqrfuL87JAq1KoJIMRhBJJJVoxKCLqgKDMiI0k+7V0ZAggakYVTcW1\nZIUFFeHRVYE2wA4i6e2TH/9qZ5rUWsMqRCuQBYLwIFwoYhGtqWrtHKAUEavn+//tH22SQJSu4d6L\nBJMM3eU2JUSth93VJImikIZIYUYTG9ITJKgUhgjbhaIicFJHVS+SkHNI0cTkNUlWTiqTmqUM6bSh\nWySEXP9h/vHEnLKATp1bAxFRghI1AiGdzhVqq9YgDXFQ++bnf3athJmGBCfN7i6pW2633W8l1lr2\nFxtR2M3jL2x1okCkIZk3OKaBou1YiqS7nwtLaAcg1L0jiQFOqhahqQFi5FzYExypyz9+eP7rJ1kM\nJF09TGBJRaUS8LEqzazV1d6O+VwKqPov/+01SCZT8RLIwRDtht1FfPdvpVLKYXsR2dHqTiVdVCgi\nghDzdjfa4yMEQwFhGIbXW1ZIMndkKZ1UmNINNLdgTknVqlIVLgwPTcvc5bv43n//mIPQYd5QhohM\nwCB1PKiryU0jJPNFCxNTef7xtRCiFowa2isrJC3lOt76vQday2FzeQgxzEjlHNRDSSgrKIh2cxCg\nmKopQKR44w5NiaGGqvQaUVyqdzXUCFVDtEEDxrFgfet0mViLvvdHsj1UB73hDsG5bTCdrs8vdqOL\nMRytSQPmfHD+k6cANWUlCKYUE0OXq5CHv/8wJ6+H66tqN1D9PO2IANsxTU4ooQ15allASBPdbkM1\nm0Qyp6ubWTvWXtW0dxc3qFX4NOX1ojdHFSol/+CzPz/AIC5gbbARQYGYHPY765KqhgBUNuRDIOnw\nyecjVUUNEh6WfILYshvH299/tytpPFyf75MREvxrBwSWqjAULpTkDCPlBginKqRuqopKSEJI0Ltk\nMCBCICYWEgyYie/Rna671guLllA9+s+efxGdKcRbdarqrYmT8iYNq2RZ2Q5IQF1ATfHNzzcgNJu4\n1NJlFtpC+7w9+uC9vlDHzfkGppjjA+aUpzOewoAqQcAYIXMzAZhvA1ANJq/qNVSFKh6wjKQSwRhZ\n63R14WePbvdRSuuQzadDvfd37h7GGnTMnz9atSjJyvWbq32hWMOctRXLKuc/v1Cz1CdSImzQUpG7\nIfHog3fXHPe6u77yJC3I0oPzICPIEBFr5VzMwJ9AVBUIxG6CKUJzDU6TdkkLcqhZUjWrEyqN4S+u\nj98+TWWsLR+5V1Mgf/tv9dN4KO7z1WnVqplm7C+utvtKaWgIRFTN0u7n36xOk2ZziFf0UpnMQq1/\n+GilnDRtLg+SGRIxt8tz+qAIxEQj5sbWbW7IVGByKEgCEXUGXbUDu05pELEpaBopOa+ms0c9a9EO\nHgGlJaPQLf/h13/mKdRgrcckRBpGVMftfhg0m5M3oJrhya8OVcLEwVqi62s1YaS8uPVoLTEtjtLl\nVaRWj8y1oAYJFWu1tYFo9S4BKjwkVK3QIAgz0hkpKxXKAhO1MuXIZtTry/7BnY7wJO7uGpqDJQRe\n0/I/nX61DctSeYNnCcBwmG8O42Rq7dAKREzOf3VdoTlrIKaSu3oQkc4837l/mmuxdUqXxQRQUuEz\n4kFECIGawxquEqpQUCVEVKlp6yatUXUzQpJCkItWiyhGgzO9vrz7aKGFfVeICs2sJRxJFR7prf/T\nz//xkwJLNwNPp5hABbx6YZ2aSoN0RMzGT76sQusUoEcedB+dQdSO7h2rhBxnSRNMWks+V9Jt3tQq\nkdB5oCekCRkNvuvKFNppoJVEkC45UoLRJCbWRGG88g/uhTOzaERSLZMkms4le9TV31j/6S8lp4gZ\nPxWNCDGbJh2GLiVtMUBU48nHW9HUAB1Il8fIAEJuPzpWOtYdkMRUqCGqDdSVGaOFAhTS22w4rE1R\nFBFHunHXZBKCQEC6HJY7cTE6QgRap3H9/plZwQFKkyroUkUOiFRVhnjJv7X/5nzVBaKh+WRtZVns\nz4ehK52AiATRy4/fqIqZOFN4Z9Okpg49fe9eioplD4S2zkYhqq0NaXCjBJUyA6UMb9h4IKLKO9w6\n5kEOgmbhZknrFGIm4bXudt1772QKUaVGHErkzjyj3kR3hirrD/8Ypbj/ZpIkMFOqlovXF9vD1E6P\n6PTp11RVNTcLF8OExHCs334wsNS+F0BSan0n6WCbezagVMGEgCkDOo9eVIKIo9NPvGsoglKSmTpE\nUzmIkVE5ufDew9sBnwitWqDZQNFI+17aNIpAEu3+6LMvpuJkGwgEW7GnPp73ncQiIQzg818cBKpG\nMAKdVsnZD3723tvLqL4eLKiSFBQoSa9t3oM2wEEQdLsZwkrSVsY73sZWQHUNE0sEkTTpoYgi1CfW\n3fL92wMWY5lyKpMtxFyYVAuyKw8iSQkEVd7/r/+vu9pOSOvhIKYuGptngltvn8rT3fHuzc9ew6Ap\nVULDxGuyKCXdf/fYJ18uDaRLEsQNeDqXUa3mMhHCJMRAhpqydYp+/N43UyJFmVhURc3E3DUHIWXH\nuj97535H38NQNtINqZXD7eGsIEyV1QGk9Nvf+zde0XgX7We2Ybpf3f+d3z+J6QjLZz/+IsTC1KG5\nqriLYfThwbfOpNR+mRokmRrHQR2MeZbXarhgNbdiEgFoeKiSDi8PF2+qQCHZoSDV1FIx2yPpbusi\nb7+j2h32sRzKPvUWruJG9ySVktFXV6qGUop0P/yL4pznmARCQbCGfuv//IeymYaTTt67szMJmHry\nERCRnKZJTj56kErkVd/SDZOEQNFg8/hrAIcXBuIGdhQq6BB4fv9QrJpYuIhRqUQQ4y51st/Xw+m7\nR4tUD3vYYZ+OELUT1xDSpjazLMaaQqQKJu2+8+6zUhviJKCER3jU/g//2+9ebxani6Ta//6fXoNJ\nnSJO9TBx6vDWo4W7LjudK+fkgoZNa8OnlUG2uY57sjbJFBCgGpxHt6/b7C2UoLpl4TQVP+p8ci93\n3rmV0mHX91pK1qqKCBd1xKS50wiNmmLKACNhsrv/yd+fAmINpCei1MqTv/Pf3Hl6uHukUsX6j979\niTKq6hhMqDC4929/cMTCRQdHgwpTYiMDtAQyl8BAhJg4tY2foPMcQaa7y1ebEko3G2GeBqtThBzr\nIcZxeuvdLnNbGT6lEz+klVUUKVkOklPipFogRUELS2Cx/Puffh4VFoogGeN+4jv/h/9UPud7C5aK\nSLZ+96eURp2QKGoqwPH796xE1zeqDyhM0igMiQ2CafgD22A3KW0eSavAlEK9E8WLAWL7cRVdzzq6\nLBbjPte9Pnprmco+8tK9Q2VOQhtFbMO81lo1xiG5ppkzAdGDr/7mqwp6GyWUcZrsB//t71w/Pnq4\niOoSPtR02s6tOKVGEoma3no0lEi9zXNMNC6JWEA6qFBIhVDVREzF5jEPADWAwoln7qIGjWtZWjZ3\ncem7QO87PHwk3FfXNPWDjHXofVQYp6lbaMAmGotqoCsOZIEwCt79k09qhJJwH7ebW3/0X90/P79z\nx2qU0aRLAFw1aymukCRSRrn73nG4DBkzKtLQMMwUEd7MXWdOj6haSyk2D/sFGI7j+mAh7nJkIGNK\nXRiYpjfy/lq6cw6r7XbZo2SyCqrGdFgsMEmGuauUzjav0pGKEK5w5N+9+7GnAMLrYfroP/kenvrD\nlVTUKjATqaOoUkRBt6zcH87evyeT9501yI3i7Yu083FD5hPY/Ep7tR2fgEKEXK8O1y5TktwFXYex\nS9JxX20j75zZdueQq7zgoTOVXenJuuNKwzBSDbAwTud+xxrQVEPI9E58AgnGNC7/5IfH1/t0Zwh3\nOFVNheVCESGmJSwJYrJH7w6M1AJWowwxklK0Eb9MwaC06ksMAMJEQFQlAsrwadG/PGguYlZCpPeU\n1GIKbNJbZxGlLPTVyRKSIkJyrYZrnnZl6ozOEOTE8Vm6o25wIQln1HTv1QYRkT54+3R8XRfrVKOC\nVEkGcP9czGpVCYeRRW9/cCJVujw3AA0KTTPECkk21/8250VADCHaJhguAbif8tL6Qw8cxPMQUweM\nhwRf3urC92l9qad2GJJXj64f9lcx5MqujNGxLrJU22yOejBmWF9kUsnd2TXp63fuxHnVkwElglCF\nqYjg8lxMLG4g1rT86FHnrjZ3HnNySDIPVAmBWCPjhYkEAEObJrSiUYIVd0JNoGCuWaLkLkYZ5LC8\np9y74UKOTIJenB3c95vTfmIANe+TFj3sMxdWgJQkxUGhGkWnWFlZ3r7THUb261RrCKDKlgPk1SaJ\niLlXzYoun721cpfOGotCxQMgUptCzzQUbZBxS/WNeKQgIkRBF5Z827WzNNE0CZCNU+6x7W6Z+rbX\ni3ysZYHzIWUiXVY500kRBRodBx6u+myyS4PR3GsxK8Kaue/uHd5ebA/W5SwlAoAJGO20vtgJG/6e\nCMurR7etRtcwNAJs2E6KOaVry9cqNyEMjCmbwxAEQmB0Pz12tWwGqdFLMmEaYpPvGWvJ5TovXZcH\nWE1d2mzHfs2oKVGqdFxy8rUQuqRI1Ni3wZXUmkTumO6Y88AKEmECIKZOSKlfHjJcNQIilvo7D1cM\ny/obeqEIqZJSCVBCwAjMI90ZEmpko5msBaiDy36TrFumulehhPcmHOVOln0sy3axiFSvgHWne62H\nRZbCzsckElO/Zc6ufuiTVmJyL51FAfZ1fZTr9dClPscYM/UJ5DVyCOPqkxCA7jRNNW49OJWKZDpX\n6aCCKkiVsPaxYx6r8oYMLjd4vIRQ1QF0cA8dwnwEY7JFlX2+ZyjM0/nxmqrbabnQvvirxa10uOot\nulKT0veWwWoWVVWLi1RU1VonnpzgUochmdZSDW10DLk4zzlT9atvjJGlJ0VR0umdvkbOAvwGw4XS\nIoXS25lubRrlJnmomoAiEg3cp5BZAysH605YrRc77E7vUaun8eJ4qFmnqdei1baLtVTqOFhV33LB\n0Fo8q/STSxQ3W4y+66fD8m461L7rknhl6MxThexe77v1lZzgLy+yqsRYwZL74wdrhHbWxg3RAAZh\nICFm4rfMRNfGT2yEaGnFGkGRkCDhlehLz0Gv9ktZlbq6naZDWpxfnh0ZK2WdfCsyHh3j4GY6ap5C\n/ZB1K10Oj6EbC8LroDj4mO4c+yYvOwv3IqKN6SIi9fVW+1ebR4urH8XCqqJ6YqxOHt7Vwk4A+Pxw\nzfkj3cBgOp93AmyMpJm26LhJKxGRnWDXHfLywHQ4HsvqLkbCLq6PeoiPqZN9v9znpUwUq1llPy7I\nzK0iCbSQg4wi4Yd+OsjpmV/r0Cfx4hKSRG/o25dv5GT88gz6688x9YtpimS1W95+Z8WQZDcMirmh\nFSJRHcJQwmaOdWPVykxGCYE3TqWHx5JVZCQhnpA59fesjildHBbHqJysMtY20nq/ln4hI2gc+3HT\nZRhQs3NyoSuqH2J1Zte66PrkpTpERVVAhaiML+T48GR1AvvZ3iJKhOaQfrh9K03I1rpJaSM2CjmX\n723CB8xdCQTU9pw1OrCSiQQcK5ecd0ncFukQ0/HdVFxxUY/WzFNJpS6GupETO4iyBMcuqdVLlwVc\nfeoKWCbv7JCmms+OxjEveoudNyhTtUlDVPF6ezy9Psvjg+3PLSurmoCpO77be5gBM+O8cR/Ixg6y\ngCi0jWnYJlMixojWBQIkKkQZstIe3eALSO+Y1mcTXLo3dTGIBmQTR71s9YghY7fYcz9aFjVZoyLC\nbDRalDSKH/T4dt1zuUIdi2unKiKmggDU9Or5MO2Wu/3Dsz97IpogsbAaabh3gkA2kNHYor95sphA\nSU7VpG2OK2yc7CDdE93QKksiIsJSCluMk+bVYdTbedpnPZflSUy5xoHdglc8yTyMo6TefG9XT/Mf\nQqLCgomjliTObbaH3b4cdckONZDSzGhHG/BKvEBcb97s335o/37Xtal7qqYnd3ONlERmztBN20EC\nqZFMoCqtVqSQqoxGDjLlzIRQCZAiJlqlUy66bhl7lXRRl2tz1Ho9LGSssub+6oVinOxscXho9s2z\n5ZB0EynW8Dcv0gcssbjHy/5W1pgqEqAmM+IMiBrON/7yq4v+wQf3n/wkgpWhMYatHhzBJamgEYjm\nNqMdl9Q+vzDsZqTdIDpFWDsviZh5VIhq7DSNldC0ttGsvzislzrC625Q9Fe2Tnz1/Pj2Ybc/fDW+\ns+4Gvn7RxfFxjmdWhxf4+NUfxoP1dVkfDSwUA2jSmqJGyTPZPL349Bn7s299aP/0RW/S6G7UW3es\nQpM2acpNxGqgLlIrP4ywpK1gDLlhFgYCotoYW6KVKEzjMmoZh5RLWKTLw2qZPLCrnapfy+CyG9+x\n6/PYbMrmXrUiq7G+/vTYMO2uNlhd/PyHH9aXdnvFSjdtmMLNjFUgWl598cWTTdcdvf/B6sf/C1IC\nPYjJTx4eo0o2URLqrQhpJDUBUxPCRDKBzlS2G9j0hi4LtGdSs280NPebcRvDRUGn17ujVZpgW/aa\nZK9HOm0vjvvXL63GyUbs6RDb5y8rHvCzfD35pOf5+/f3V7dPevFG1poJdGAkiHH34rPHl7va5fW3\nvnV2/aebZIqQ3mKS07vJo8szkUhJBEWlsSAlhRprowQFVdjKFxUEVQAa6xyYNYRPiwky6y4O7Fh1\nv1hmB67rUuGQBWp9tegvr05kczd8fPPqrjzZXS3vb68l191k9uhPvrOp751pdW+P9w03HjDqxee/\nerEV025x561vnbz5J38pKQEG2iGWd9cM69LMUwu5IV8qGCJJldAqmVCyNVttkjDrY0KFUBcJknhV\ncgkzjuuFXsthHFYLSOxLhtKnhdZ6sVzuL/puf/j8auubo323snuc9ofoQzA++tvf+ubknaHc3OiW\nckkzqI6//PdvJkROeXn/nXsnv/h/f1EtM4hkgNy6lyOG1C51OyVzxwEImFL7+6KNpUIxD9xFlJwF\nVC1eByxdXJ0JUsXxkcTyep+HgcJtdKZR6zJxetUttI7lab3e7RTDWe/bssT6dd4sD7v+e//V0ce3\nP8iH0TqGOE2aOksYMj198rM3apGWt28d3bp1/OLvfp6ytUNgav1bJwjtcytgb6DdebINiaQgqRKz\ncgWgtxJZSQUkGmmZKiph22/u7TzyopauO2xTvxTWfR0ImabVMHGc7vLy1YvSd1GqDB8syhfPh/Xe\njp4cDgW/9d/Y14/etX1RcyJcZ02GiMbTXz+9czQWOzq5dTT0Q//4//llJ5pIhJgxHz8cwC7JTICc\nw5ZIkzgEkwQkRzCkac1k/rYKIGpuLza9BdTik9+V4PF2FxHbWB0nxnZcKKNOKRfRXvni8Ru7e3gy\nMeTe2e6bJ/4t7m599upWf+uD/2y6+tbbXqoYy41yh6JifPnLn+pKT1ZYHPVJIh+d/99/oWI5EAaI\nT3bvRDw6I+SG33oTe9txSNq4qdKY+CJo4kgC0eA5GueGXsXsyWYhsjiVq/7JdjVk8fPDsVHrPvJ5\nXZ3Qn3yxWa9fvOGw0+WD5xdf5+8++Gr5yVZXh7/xuxfjh29xcioqqVQqRETs+if/qt53390/VVEi\ndWe7/9fPRbUpc0y55/rBMiLn1k3NQ9s2ppVWzafQxiq9uT43z54CUCFnIq4YVWDd7vH3OaGu08Xk\nyyw+lpWhwHicuDm/HK6f4VRejHa0G9ePps8meevo+v6vx2/99O4Pbz2X753oWN0k6HB2AtAkvvoH\n5wt5eImJhInY8mj33/1FG5FBEDCa3j6Rit6g0bBpypxHMLMc0gwBULz6DJq2HivCm8AlZlmuKtDv\n//1HPa6nI+5e3slFpst1Rq1Wl8m68eKiinXddZke+TM5Sk8v0mrx+N7T8//48fD9O48X37vrhcpG\nxFSGqjDq5//sk5O7l1cne276IWtaLn/5px9HsiSicIggsLjT1dp12vQqwMyRm8VBJKGmqqYRTSiB\nOYc0FggpgInUaDQo0dw9/onlRa6rtFprHrddNkClz1fPX4/77bRb2uuL6+XbF55OpnNmfP16fHJ8\n9+qH+OXp79wmPSBUrSXMAIaXMumj04vl65p4WZ151f+r/8vPIaoBM1YQIt3xMfc1pyY28SY2Upln\nBwDYivkQgUclIigioDb+SRP4SKMdqwjEMv/sada7y3p1r1S4DibZtOoEi3xn4Yvl5etL3LO9dV68\nw26DVO7s/vC9L9bfWdOjVnfCK5Oq+XSIvHj4XuYadr1IUm19dvgHf++VStd11sR/YqqL07Uf2Nus\nLpxx0Jkg38CeBGGADW8JKABqC2wMtqIXSklNJ0X6+V/dNaY3dXWlKEvDJFOXNkybJ2PG8Xo7ZZf+\nuizvvhasjliFZyfl/uU7H+US5uEhrHWWdFnKORYP3rzaPrx+vTgu0eHn/8snNUmkJFAiwtQ8dWe5\nRp/niKoz2tNocmw6igShhEgwAoJI87dtoulZoTKP2UXMLdfPD0dX6G9dW9lnkXEni+Spf/P4Wscl\nYyuah+v+VtrtVFhsHC66zXa5eNvqqOZTNXUVoHruVUQlre6/vLq4fnD4/AeZuyc//2Q0agbEBO6U\nFJFPTlhTbwRbWzE/KMKYvwZFZ0Z0E1X5zN8Qoc76NtWmcZcmtlDTacwLXXBKQM+xDGcr2b2aPGTa\nvNjvok9peSkf2rkwyv78+ird90lWVA+ph+KgaSCmtBxMVYK2PFnmiy+G/XOkshsFQeRMMWFFUMKG\nWwt6Tg2zhvA3lgO/GXsoUtPJNqkWbwSAgIIiYjOhEI0MPLNRIH15/jSLHqUNj1bwst/sXvb5aoxs\nsVo9wXh4b/eUSkeRW+9cLe6/fbYS7eRQPDFhXzs5WonOagJZLUxeLLuL+7VL21CqUBCqFapk2NGZ\n1bTIrgLe4KRsLXn7BRuooE186IQg5jcz1NgwuoYJN+CLaoYKmT7fnAR0qqsh4oDxs+CDqz10f8fq\nEkxPH71/eVGKV7z7e/v8fXYr06TTNFFMp4PaoleIBiURfZ8NfPqoVE310lqyVk0OD00e3dlaOfTO\nbBCJxlxpeXCm7kubUYTfqNBxIwBsQ2nOIlwFxUTgFNXkFypPznVfYipHi/35OFatu7P9NYF6nqZx\nKf3063Tv0Wkt/vA/W+sfPIpdck0YJ9L62NuQTRgUzcly16/WXfLDftot+nHn82wAjEII2J3czm4r\nq9NhdALzxEZuQOz2lCXX/0BBby+KQERNbzqq30h8bKjqlstXP+AL9MSkp6zlefZNZDl9PKWevJJh\nq4vDapzefIVa7v7tvPv9W7vFfiN5KlUMuVzqIiczo4JiCebro2Ghfnn68qMMGjq2XrUGJYn3t061\n9gslvSYx+Q/S3/ZQtUvfBI1AzLepvUEZM59Z1MxEwaj29nFQxLrn17FN3eB1aVrfHF68fnrtne11\nOaTA5rIe7Pb9w/P1uI/T//zo4lt3Jul6uKm7dd3hFTuzdFO8KiGpP151AzeH11dYLbOGqKgZg5Iy\nws46l0WnKkCt4ziVCCJ8pjHMQppk80kndZbHsvEBBWpmqd0qUmVdzo0Klc2vfr9Tl7SU0q0OGi8O\n2+lWGdPyQHZ1FD/c083lu/3V0Z88Ov/wLrraFQxJA4zxkAaVnCAQ2sxC6Ja9Za+7xTVSv+1qE6ZF\nAKDb6bGWbpGqImZJcoXYLEhv/wZC2zGnhzFmBQQEFEHK1jhiToqqlQvCzDTh17s7KaZ1ThrnmxeP\n62mNfRwvrxEmWNVxef5ykW4v0h9++OrWfe+GIfOAzqcou7HrTRWtpZKms0x5MSx6kymuNJYsZjfw\nvym9f3gCdkmsy6lBofRpGvdTrc2egiQpiSCVHig3Zg7a0gwjTOEGyIw3docmDu22vzIbV3012b1+\nfB3vXdU0Hs6edUevrGg3YVv7J/Xu2cMfXB7fPXSSxPcnQ706wKO3XhURsxYZAFXysB6Ww1gP52WR\nHA2GhgtEC1Z3uiJ9QlMYBwm6hjOm2RKBoiBTSOPBRosGmAmNTdgzS7KFEFGflBVA5OEXuQyrJFN9\n9fVOr6b9Wndf3x9ubYqWoxjq8/Xtuv3lb727T2+FG2hLsXS5mURMk0J1rrd1Jr7YsBpWw+S1bgfv\nD00MWScmZenfWofnTqIx3JxsslhRDwqSaGON36ifm7xtdvS4cRD4DYOraZ9cFRE1hv3jPbLXsU5X\n17e9pKEvvezftSsXX+3rWZc6xXa4mD6QiCkkFvcXm22xvjNTVUi4A+5OujPIYZWTupT9drGmWeO0\nBonA8nZfIjfisRCWcjZtGBwYXqYxwmMeAjdDBrY8CswVJRlzNmxsbGUg5a5fnAx3P3y/VHcvL7uT\nEd3JZeWDd+v5pEh5Jyf16HNd/0F3/e3eBgmI6Im+eb4nNCdTUzJqjaZ/igBhXZ4c8N2mWyz6ZRJh\nFEmmnk7X6uz0NzwMVTVVM5nHB+HTWGowga2YVxIebPVv61hUSaoghBRtAiPw/tn5Zy8/f3dFP+Cb\n7f3DfrV6VpadP52uPbHbTUMkcf/++49/+9Z+NVWDme9/8tnyZLUYFktVZcBFE1itEWLqftqfT8Ja\nr9NKpVJS3gdDyf72MqLLmL0W5rgqphFNtC5BBIokgopZQD63Vb/5RzDXmRpChgkE+tHLf/OL/fLu\n78WPXu++Pj3+Wm7d/gwsG8/o1Ff9Wuvw+s2j33n+8NG0WB58SCoXP356hDevDsPJ/TunFsICCzho\nYvDp4skXzw9OQi70BN6PglqaHHaxzhH9PFcjJFQEUkUU2hSmgiAdSefm/WbWQ2Ez8oj2B1MjApJK\nhErqXm8PSY/vbF6+fsm0qherW7vs9ThNaX0oNW7f455Xqz/y9XemfoW9rTKffpp+gNifP37y9Sf3\nPnhnoQFvKgWX2Jw/+frNVI6iUvYbXy+ufQiMVR1KW57oaL01qLox+G8+GhLgGkGp4UggFN6UIwz+\nRjIpYm1qJSRDRcEpYXGqr/t+aXX6+pPww/HwJd4v39z6ouuvsDr9ZpLxyfv6Tdn/jbfOf2Ac+vPp\n6Eg//czW2Nr9vn9xufvm/PKjNU08FIBPrz/5Zop872Qn9VAjqqzgKQzRuG5HQ/gqs+noyVnMqwyB\nUqBI9BBKJDdRSoR7mym0g66i6g0E9tbj0wHpTo+qHP/xZXxdBn9zNdzS7vjsR+m31qH7cXj0jMl2\nn9yKVw9/8PJby2m9iGl5Kn/+acaVnV//wcudP9zRt6+QUnRirLE//+KVLq5jezn6cgorzgElN/IC\nPI5u55BOZ0efNr6RuRUJgSghuYYok4LePKva8NMoZDRfLFcLl5lJAD0+Giy0462VvvjReFit7p9+\nvFqWQ7pcbjrE9f7tncOmx133Bzy+e+iW3T4v8U9/gVoO3NerSz9+/3RZUDarYCgMcXh2fvl8t7dO\nMouHBKUfpsRaKK4x3TmzyX6j7wWhQGAOp0LQg0JNzkRpJkSM6oEmgAl4c/9inXVQ4KQPvrP+QqrJ\n+M0XJ19+GR/eOdw5pL7Pmq6kz5DtF7/XfXKdhNfff2f37ZL6XH1R//GP6zSKR+SdXW+ufu80hfjY\nZyCZvfnlXz4/0EqlFNsLWErGskREhahFDCtg0c1MZBGRkHa9Z9V4BKNZLKWiAQRqKbXyBn8kiahK\nwGFBStTyzt+49eRgKtf19ZOvz+Pb7365enpxGyf72k/jermIevmr73745LzWk9+7/LDb9cmK1X/+\ns+k6FFG7W3+z+3efXv2kexgMrwHr7fJf/ctrWd966/Nvxj4OUEr1XJBtqhQguLjdwQbjDJ4E7MaS\nRQMMDY8GZAWTtijmDG/Rl2g+WBC1ZCpBkfDg6qg+F9Hy/GQnr+zte7+u61/07yqem2/r9OBkb/L6\nr+L4+Dx9b+EPDoN1ovyzX+MwLW5d5ZfHi/c/eu8ff3r76tQHdwfzUH/8r6+rPXz43dXluRoAWKmq\nQ1EAqphw606qKZOzzQAQFG/jkea34oymdkBqUp2YByIziKdCACZmMuvocPX5R5vbxzse4urTK763\n+oWcvra+366Keeh0effNVmU/Hoq99e7l749Tv+iDP/t6vRddvdVr33tef4f1yDeqpkm6nB5/PBjK\n/s3y7q1NrRpJ8hQL6WsJFyHQna0ZXSO9NapJS2xEtH8j6F4R0CFpSDN2iWaoheZEBklZRSRJaGi/\nvXy9efr8d753+OrPP67X+nDxPG4d0uliVY7Es0m5OGZfKTgI7l/f98tl10f9+NdncnTxeeS3vzL8\n3oeqJ+9eLkuuTrXc+xN8/xuN6e2BySrIscv72iG6yWGSCnGyCOTZP65VHLNDA7w5dzi8BNJitUz8\na/3iLDWe0UZTUxNTcH94nk4eHN97eHf7p19dV7n/zgUEq5QMOznuA9Ddl+wqGLX78L3dd6fT9VLH\nX/56SIuzM7mM0+9v3v695Ps3t6EdE9yl7w+XZfHRC3nrh3VfpaOUnAR7y8yoASSwP5GwrLMeD6q1\nAVitp40gIxzDYjkMw8wTaEyBuWAHRMxEVSFJmacfjae/ff/ktE//4h/+4uGD8dHdePw03YvusB60\nu/d6v6nVdhbJnOy+e/jeyWXf2/jjXx91YkP+9vlYVmen56NGfxRjmQQOQHj8oB5/186unn913uc9\nmY0+5fUrFp0EQH93Bc8WIQwJCDrU9nQxPCQqhMyro6FPKSWloZK1yZUbxE21hlmbqkjnh0c//M64\n3d766b+4jP33p1ff9M9jsbxMm0Xcjf2jFxfIDDL76PhoOT7YLPqu/tXPjvpODHrU8XJ/cXg96qPV\nMV54p0CUArut12W3fX355WFY1H1U6yMmOYrkbhDF+v7gyBZtYkhy8ibeDpdwMii2PFoNnQokkVKD\njviNoVLT/kmbxlHicPxHD9bj4Zs3+Wo/8dXz35pubR++qpPkVM+P6t3rnyZoUAB6ffDDq/cmzR3/\n6t8dDSknsfCFIg7Z8oKbcJOO4SU8UkeUap6WeeC1RLiQLkdaKumrqOk41dTNjbkS4b+ZVEUgQtif\nHC06E2kAnbt70IvHDU5EoWizA2Rwyr+38IsvPysn5bqsx8cn357s7PVhk7YfbJ5D7dXJbgdQWSu7\nH6bhERfL/sf/sl+opU6ZQF9iqSl1KSI2XYpABUtanIilute7k9QxT6LujEgyUpR2qP0CTBpkm5Vz\nBnHDmxorZH37uE/N209SeGWt4aVMjoZk4wY3mWv4y6G+vHo83Yo3vjgrd/Rn48mznR3Ong/vTq+u\n02IsQjKMpm+/9+q9Pi3Wn/0TW5uRopoFrALLKkFeo7OA0GtIGg5VxyrHB47ZTNUQFZ0ipHIq6bij\nZhB6Y24jbGRqEvSiJ3dPBuVM9UvVnR4MlxYdGlyNG2sRQNReyoD3Vq9+NeaTk3xvvDj2c++6h93T\n9Vntv/XpQUzc3ZSL3y7DQxv6l39/87BLUIZZgvRRYRahUdzMKYzCEENfqmZE5zWBovRp4kIPU4YG\nF6fWZJuz69o8iSPhZHU9fXA0GJr4TyQVkwqp/hvLxeYt0LAtVukXi4FHQ7e7flIWPEl9PbynPzsI\nNy9vH9eX7558NX3/6uXWAjXqhw+efrTquul//PJ+JwJPACOLmEmQGVNtg5bwUlwSkSQgiZKSwFTh\nxbUfa8dEOTq1yKmZkalL+yrNwJYROL53PFibjohCUgjnQw1rhdYMNkRIxLA+W6jj8s+/XPDSkPua\nxnuHj6+HGvtvpvXtxSeP+beOv7pokv3uO9Nwrx+6//Vf3+qTNWTcKCbqpvOPSM3RtRbSpDAZSNUE\nipm57B2ngQqpOFkC3exFGE37SYBsE5Dl3eOuiakacJ1qQaVGqyFlngmx+fjChu76cjEd3pwv+wv1\n5d3ro7N0/uKVRaL4a7w4ev7yLWTJAQrrB/cu3llb+vhPkftkCCIsyCYlhsA9VNisxzwkSbioeUBR\nS2hdOad6OIkqmZGOO6SsTUMFidkiEQQQ3t867mXmb4iqaQonwwnOxnWNwtF8EVTGYjnO38SJ7I9e\nrd7ZXTzqn3+6yPtIhB4usVMg+NYzOq3kb0/5vYVe/A8Xq2xCA6takWRKMS9keAOYyMTqsDafVAF8\nAuEWPvp0lGCLAxbHitw11VJzwG3aOPEI5/HpYK2ZVZoaRD1qsFaPkGgGDO7RNHCgOxm7i+MPz3Yn\nq9sfbL6oF7UuNozOVFynaatp8Ne3vp28TvWtW9d3l7n7579etElrFqLSx3FyVgD08FKqh5PiB7cb\n+jgYY1R3kJPlfs1QseNjcLGYDbFnS9iGkbpzebZMM+SmXU6qojUajdQjmjmRewNTGTGs3AP5vXd3\nj4/T9Ojl01V+VoaTfVokcfcp3d3WznX3zXf+5gBZfLS3d4bV5/+09kr31lurwj3cI1rvwDo1J8qy\nc9Ua0aSOZRpFsnjsJ8Gt5E69NZCpmQoSjfwXLaRGpLOjGXuEppSarrOVlQxHmyeS0pzuoi6Pqt1+\n79H4z3908u3x7OnnJ7f7+8fD2aJ2xlKc5F07vnXnvU9/8dv3w8/u7+/cG7b/4KUOQq8VgDcHoupR\nahMbl+oejChOlw4EaxCY9kWzqaLsvK4thdtZRrYmBZ2l6cGmuCRXp0NrmSTllEwYVIiaoFWSLSrM\nOccZZXn0ztnVm8+v3nnrWX75cnl8dHx/L/5AJKKW0t97a82799/6aPXzr+oU31F92Pk//AtBZy0l\nNfPnYByKuzOaJ4YH6cE66hJtdICyn4LaQ6XuU17VKHV9bMgSsx33bDeLBk92p+vmd6Y5t+gooqJs\nPVZMjfqnwjnK8GJ8MH69KXn98PzpJi3uyQDZnhyGwyYg2vdu9fb97kU+e/WTFzh7tD19lH/0D12k\nbxBnI/G4e/ihVC9k1LmkC9bDZIM28npM+50IxcPLLklnmOx0SSSQXhu1dwagKaCsT7rW71lKAlJU\nNYkEIeLu0boXRZvdqkh9fDJ0CwfflIXvjpHHusnLaw0PGVy0e5K/h28+++KL8fF4/KHVd4fP/4eL\nHLY4SES4AozaRfHZJbiUqAfttRFWJySrUEawXE4KUxerG2J1fOHHp71oQhPTU1v2plDCPZ+umgIx\n5dS8/ZSiEmoGJyN8Bh9ETSlSubi1LLv9+fXJdjrdZ98cfzXaXvq4TVksfIzD0dG61+XTr30c/uZ3\nr1y/+NlrUDXZbIFLULxhHR6gR+wPmtKiMzhIw+gREeVwRSMNIeUSeXUC7291lmx2eg0PmHAm9WG5\n7pUQWEr4zXRKLQsYagpV/f+Bffvb76y3o/oGNd0rUbYLXmK5q8uru/1xHBaM4d75P381fPGrAw6/\n+ycfvHdrj2cbTSo2hJrOHo9RAwIGPUBUZLN+KYcS3JeqrBFRX20aWSQSr2GykuiPFTbzdluomge1\nDFsPSgBq1iKaCMCkRSDmltKQ5q63+Z0vpHszSWdWXRf5UF+tvvVaN47dg8/f/nobw+Jq9/r55en2\nel+tph+s0x2r//bfj6YVkNT0142+BkSkmaSgne5f24p+uN5vDnuQgrh61ty5a6R6XXofkNYLYScN\n0BIA0czNg2S/zkJCkiFk5p4JUmFQFJKGxZB1HiJCoHJZh35ArYdFqstaNu8uf3nrclpNjzKH67Ph\noIeXx3d+5+h/mvpa33vQL7T+i3+0VbOg9E1RaYA3ckvjR6raYorzn/d30xKX1+s3ENLpz7cGlQyI\n1c3YxUKP73aiM7kPITPSBgAhfW+AiJqBaJNzCaRGDVLRvOqzNfE6RIRTFVOoVs0yPt9cHr//tN65\nYtrWu1x9ZfVbJ59Mb76Vf8IT+slH11ZXX/6Lq6Uy1DJLSrgRbczaFBWIpMU0Ln83yWbqyrQrJjUC\n21cqVcQEIrbZHdvCjm4ZZ0NymSuTYDOftkU2QjSl1soqgQBS89kMZBs6VaG30n9m3FM7r8/vyLOL\n8Q8vn613PXy8ePdwizF9lC+/6FbPnwxvHa1OD9+8+k75ey/ySX8xUhP2qtLot67aSGAigKE/LqU3\nTen+naNSQ+BRn10DKtZpEWDaaKTFcAwmnTtWSggJYQDB1ETT1rC3xg0gNIEu5hbo+xaUwbkAEkFE\n1JMn52ud/AfyU8vlQUb31Qf68GySi/fOvjnbvHp7fXL3g6uPcfbof3486CTaTdRmwE0EQkJaLwEK\nTEW6MfqTKu+vSqEHg9vnIEWtK6aLmC4ZZXkyQIw+j2njpv2OCOTOhDMzown2JRoAx8SAh2adB26q\nzYscEOHqdNGLL945/dl+Wby62Xb04w+/GV6V3X39+q3v59N7+c/3b//g43/Z51AmmhbLIqbSHuS2\nU4ZQgCGKXtaP4kJbWU/Gi41FSA/bq4jxkonHdyxEG1NZwHmc2TzeuiSY3bE4g3YKQaqOSRglhTY3\nVja5RUDCyfXxq3GF9M7hF5eIIldjn+3FEe5d3fEXY43hd99Op+ufnOP4zd+7TlTkLmotS8NNWCSa\n9yLgKlEFYnl17KOqIYLU/ddVCLFgQGuNi5L1+EjDGvMNpMzBt+X2lBRipjcSknnErQG4u6vedIez\npZnAg7649boU7d6SH10fLc1tGg/TxWHhR/l+/Wp7OPvesSzPNl/i9OTvP7WOXqHOMobUmL2+AqBE\nLR7uAbpo7vq+T0htqMvnF4TAmGaos4yVy0WbbszkcBHMO0MikJJAzf4D2N7Kx8RmIC4pamnsskZ2\nEYYi37s8LMbhtHtSNGtU6aZ6wIs75G+t/81m8dsL28vKf3SVv/PnH/drVxF0ubCETI1C2CAZJ4op\nvdHXzZKoqaqUChy+qaoh5p1rcajUaqZdmEHABksLtXk8s3oyFTFr00FSdPbhVx9LU0gzbgibpu2u\nhDysbzBBXu1f51SKLRe9yPXx7mPK/S+fy7d/+PsfPnq4+smz/O3Lfxfre3QwrA/WylJmuE+aeSHq\n6HVy91Dl9vn1EZ+d+3ioePGa2onQFOy6HL6bUs4zx6oV78obnhxUoAKzZvM+sxsEECZ3ACKOpi29\nGU5LUHi2+CqSjK/vWzL1CcPpYXEY/f2vfvJwOH5wdH9855bIX/7SH5393Z0gVTIliggKpmkglC6U\npISUMbIlEjGhet+v9jvji77ff1FMC0xS7ZPuRWUae1nkuU+/kbXNHQkYyUTNbr4HblZDSNpD6ox4\nikJtXp1ARr398Pn+1t3XY9aPLWk3bvPprubY5vfq7gm/+5FGXugvfiqr7/zrZx02V4Anlg5k0Vrd\ns1Y4jZKol19iseiFGMNpi8Wb55tjwbG9eAloEUnEpBy7YV+3x8OiaV0bGZNsIF1zEtOk1k5I21mB\nmcyUAioS1NQ16GX2BQvi6Nu7V/39YcfT/evlEZEVK3uVby0vjz76+KXfqbx1Olz8dNf/4OJHOdU4\ndFXFS29edECpCG9eOp4l5LTst6nLuXoRYfAoL/q03H9yEGOodONSwLq067h+e9kpVJrCQkmwuTE3\nNytLZtoOj9xEJ0EkKsI1Us4z56GtbwAW741f8Gx4degd2a/6nIY4Hq+WeZ3rp123ePdk92Z7+ovX\n/XdO/x9TR6AKLEmtvdMn8VqF1Ggrc+S4n6oHcqd9n21Y65vn2iE+f6qiAU1eM0WSinJvKTVjACoh\nGjMfsUGgmlJjKTa795ZeKJxrOgQQPgtHAajq29tf1PXtzWXkeOuzSyyGw2pcPX95di/vpue37t06\nNo4X+58u3n/nHz/rBMSYtOa22wxWuawz+qrqKj2jSRnT0dFSuSQ15fr6V0Uy3BM3uT/YQSSJuKW2\n+KSJqWltR1PTfFNTSvPpnxlYDRNN6gpITSIRc6lNQTqZnv1sv7yzfFwu/OFpAU6ls9tI692b/OAS\n7usFx9t3/ln61t1/9RfwrKkGc4lEJqmJU65FRKgQFjWjZlYX47Qt1zZBt6nD9MWb5vIJCxiT9aEJ\njm4eBc3Rkwr/zTogSUn1N/7jIo0lACR3mVdR1LnZEoj6fvvsoEdvnx+63ebs+pBt9WrQ+4frD785\nfsK3j/synW7LsOe33/3pn8Ng0kkiewigCRYs+148JQWTBMNETSQcvp2SFUffYXrxJAzmXSjVooZa\nNY2AEdqK2rY2gGhWWQyGmFlLdwoQDVUBJAldLSiZ1b3mG0bB4RCI2/Eqb67z0YvAQjbbB3fs5cP3\n0uUz/d7kb5bMA9578OwfbUUECGpFFmoVHbts02GriQRTwNvQm6bhtcDA/qSf+OaTHaoWSaiVKSWR\nEDSWHnlTfAhSOHR2b/XazNdaxmNE+HztU4VGuAwLGL3GPDmlmoT224oX0+nhSn3YYru7XU7Pv3U4\nSv22ny5WdViVk83/+MoYRokI6tRFiFidaJXTvk8UYTQnb0KF0sNDrevzVOPzcxfRJAYPYW5HIrEq\nGnX5ZqQpcz0viMYumWsoRrSmECJI2jamLBeRNIIMoQZUVBWHjJ2su6cXWtKoA2R7+kYWq9uvvn4v\nv3mgtrh//d9/LgiKhog6fEpkssMYXT1AudCYt8C5wgiR3lIKAu72+ovMMIFGW/chgQSQU3TVE2al\nEdEMk9lgstl1hpBGtJkXBhGp0Ra7fgZZm14/grRULxaL0U8X17Vqvx/73auL3fr6rB/3fnHrbXl1\n94B/9hdtLq9gEOoTM4Vwqxi1rhWCZO1qKixppEW2GCNo8eXVaVEVzVLU0RkUlphJsaDMFPmY46zf\nRCezxpNjtPq9DRXbdkqBqqiwrXyJmVgTlOm6urtADRYFw2ZzsS2Hxeoudvn06OLq61/9SwIekI4h\nSld6jQIjK8KuXl5uDuOhCINQs9Qtl11KkK5LSPvHJlSkzCgDJKuZOqYeRDSGIG/KEgUQwaCKDJ0C\nwYjm1tGgTNIToBUqMTXjMNxQigQU1HI17c7A4VCZvA9//tWDD892x7br0mq1++Zx75PBxaTtxMvO\nmhEmZCrI1+XYeyZLafY0yuYSIZnF7cVFF2YB2hQRpkmEnbpJlRvPTr9RrDdLTAQpi6xSI9p6OsaM\nuUZNIhYED11emNxoMmS2GLDz1xpdit5Hj8NhwX578QKyWL7envhyevGNjRCE0JVqArEKN0oYlKWz\nba1HzEZCqkjUcRBVCqLi8Ma1dBJgOsikmgJ0E9EkApWo/6HXAOntu7inwaRpWVsnHqCHT9evkkZh\nyLRbDVmgMwUYCrCG4KLKyb5KWiJCLuL4ztVifzng7N51uXvns6OzJ5SAimUqBBqqEtVUPQfZr6/L\nRXXf5lXfJU+gxCDwkFrrYRKKwABTq6bGRJNUTZTVa7EkUHgbMLdJOUF2q7YlSjhXhOHj5Ysvv36a\niigYPnrsVwttaDwYUb3c+t0X79U3q822c5sK9VBO7jzP593JxX5djtPddf/69YVBjLk7lC5LqFGi\nQkwSrfRLjIzrXR6Ww2KVB++kovepwHd1v1O3edcKyY5iojCL3FvU3db6pOnGyxpCDac7V0uFB4Qz\nsl92V09+9asXUyQVcbqzTnW9nknxSvdpeue/2P3q/c/K5uiQXTfOyu3th5vL8uad7vF7E2L/5sGH\n27+8ciW6PDKCqCpURiSpgjEfrbr+OOVFn43ZLKWsEil32C9U16sizUROiwRcYBIR1Xr1/fUUU6dm\nzCaaWwCnu+No0HBv3FnGeLj6+uNfvzzE0KWspFNjgvo8lSfIGu/80fm/un5QbbzVR180F1Vc9bde\np1e/Wp++9esvjhevfv7Dv3Hnf32ZTB3ui7YqMEB6BbWWfbdYn6z63jSbw0QTEJOL6fEw6Adf/+rL\nc1di9JpEGsVVUtSOvj9oZTFRMbXUKYgQiaAdd/B6M+icthePf/TxNWw4sZQNRPU60mo4eePG/OCD\nL/+3bf/kspbUbXPtzqrL9vTq+J0f7/fv3tlv9clwq/z07Q/+9v96gawWKSfRCCMgiOSguuqwXHRS\nlSoCE2/C4NQn2LB++w8e/+Uv34zwICR1YFW6ptrBS0pWWBuYlVJSFUGQtEHpzeOfXqfD+Wdf7JCX\nR4tId8oUqkVin7eHRZAMjdA7Jz/+3638zmcjxyKTVj2ZLuMLeevZybfvnN4++tHm5PF9Tnevdt3f\n+l9GZVSGmrB2uSTQs9IEYTnN5DxRiQgpoIhZUpCy/O77T37608dFUAxGhIdE2CA1loZSo7I6IMim\nMJHw2jXyhwgB3++e/fKnr3Q9rJYS6aPN5fXuoN1U7OL0aDamGYby2S989cHwGFpeHfW+B09K7H58\nZ/Gk3D86/uLVka93tvzg/U9/8bs/+HcleYWpa8ATvGnkqKVQ1SOBLgmhMSdfVWmWZug+eO8P//2f\nPT0w12lyAbyW0oVbgvSHqF6kumPU5FDVMom7xkwyicPX/+aTbV6mYRB06ez4+PXmcsp7j3FsdTyt\n370+19MPPvifL0E977/9q30eT7f9i5d/9Qf3v3r2Rj+5lQOD3x6vr7c/+vDheUu1rVBL7ppcu1I4\neRQkl1kwAhfQNCVTE6ckAvruW7/9T/780hFkoMIFyTmkKtpFiuRjserTFExLmc5jF6SoBLxsn/3F\nT3mSTXqDReoj58u0Z0weQWGIMjZjyOLu2fVjEUHSHnVaLPT4Rffii+O3N6uLfXpbduOD6+6be9+8\nfPd3/ndMHiKmEsEuRjrNNIxA81eXCDORQJhYSiYRCJUwReTvPnj/T1+U66OoQsA5sFoWSnJl8lRr\nnaao3q+kxAiBiFFievXNv/8YZ2tQcwoghebcdZejX9cm/gaVdJdv68uvJmGN9XtHHwy37n9d33j/\nrfqzd9+6+NymF28d6TLixZMPXvz8j995QslJTUAlUg0k8dVBISEKaNA0KQCfd6yCDKGLiND19L++\n+/95UkevAolJFo6sRpq0PUSWLLvUxQDR9VJnw9vdr//NJ7i/pFuftIyWSp8X2exymw6uKmYENOBn\n3/t3z64WexHdfL4+/r5+vh3L0QdHf2W3v3wRclIvbq+nbLf+Sfcn//Txd5+WmHlTpARAMx2GMQhB\nsvCu/433nWaDFxMVhM3aye4/Xvzdz3fTpDlYkrW1yAJRVoWoaHIbs0GHh8dJFKRPb379Ge7dYmE3\nmENTCqcNAm5ym2/pDBDePd7i0rIyFvw0vb07uB6tp1+8+e7xX3A9be/tqhoW9+998kd/+Ozts8eN\nZQgTKEK097zA5TiJweIwZGtWAapmIgArVGEMihKk/R7+u714LCDshhDO3pGwgAlQs2om8sn9ZdvE\nUK5fPJ1unwhMFwMrUq/bbanarW+f9rNeWkSEUHkzfbS58tQPS13afuMre/OrX4394eJkxOD79eHS\nePX6d+988x19+iE0Zc1ZQYVlE9M0rBdRLNs49UsFTDVEZl1jePVa287dAEn93f+jbnaTjxJdHhtY\nFWwwm5qYWKfQ/uR2rwAQ4/mzzfHtnikNq4UKTfWz17tJF6frtc1fQlRVRa4fP//pKDiMB25TfXmd\nH2at9XT9zWcPji6rXGA5Idxu333y2btPP0zl4MGYSnHCx8OBtlj27s6prhZtw703W/dg1CBqePWY\niRZk/pv/uUy1HPaHwXxWrmHeJKoqYrnrLJ2cJoUIY//yGW6tiEjL5DWsz/rVi+tJuy6vVvDmKt2e\nz475GUDn0O8ncx4P5aSrfhfXm1sLZLxQO7+I9a13tn/18OTWyeTFG37DKhKMoesWiLLDuv9rw2U0\n0k5UJ0Ghx40ocvFf/q7GdPBYwxGOZvgjQm1uaGJqeTUkFTDq9smbYS3o8mLQw8QuQa9evp5MU+4X\nKNTG0gKBLYd7DghOxlI12SIfSopX/dlhvL6fjk8Or3A+1V13tvv63551pxEeIhER2dDpFINojsm1\nN7ZFXqLajOGaypwRcwtIMkjc/S9PiniJZY0bG9L23DXAncF8fJabM1A9f1oWVizlHrudp2Si9fzZ\nSNOuT7GvnJV8qnb+5vYVEmzwS3J0tUQudfPl3ePY7Y/99vu43PZd2r/0/vm9o7MkCssQk6BXRp9L\nNEcfcao4G0qu0mhu4cRcwM/s5OBv/cfY1j0Xsx8A5qawwQzuXN5992FHSITvXmz7npLzoPtdRTIP\nRT1UiqTs1WeksV29N1NRpdomlBGyxoEJaXt+tkqXe9JW4353ela3T07ldBg9kncZgPhU0clgdd91\nBp2LvGYGJdpoem3Ticus9RABydWfnExl0mXD0We/dDaCkYee3b17a50VAKfXL6TP0plpKa5JI0Sh\nZVdgyjHa9vbZwjQPg/bUzl0oWYd7Xq+7tMi7x18ftOg4FgS9lLvr/Y77V0h9ajsJlV7ZrXVMvSoj\nqSjUQOfs9NHISq3OnlkAbajz7R+WWnNHAUwhzaYX7XvL6cN7x0NOIkLfPL009dSlJKUgZaltf3mp\nhVGN3kR9ApjQ3rq60Nqn0d1lqmernUyHtMji59OI0RcDdxpvUL8jxT8pGZ34QUBxS7QkvjxKsASl\nWCMHxbyVua2D9gg45WZPPcO5+uMVo+9KxrzXPQgPegTD7j08G+Z5yPT65SRksmwxhSQjDBplN3nA\nsgrrjewtCExPJvTdodTQsu++7dX82gxTzdiNu82Ay11Ji/z40R/8YPHq79wl3Ktk4zT2XVqtj1ei\n0qlAk7BtSuZMw229dNtN0SAbJcCQ33o4+tLcbu5ZAAa6QzjcO150SQFh2by4VDTH7OJhWaAcVeiH\nogLtrC3QbSpXie15dIv9BIhMy9/S5288yhCl7mTwsn15OIqjoa665UX6rsjofd2Papos3HN/dtIJ\nqvZiEW1bb4M8m4+J3CxxFrZE4u1WnX1PuGyfoRmpz8p6r1yuOxNLAsZ0/nqytg99PKDt7oopURGE\ninRp1ivPNco2r9OmiILsPrS/+iD6yZZjVSI7re4W38VQ8fHR2fLBX/7j3ck6XJ0eFoFhue4k4hCJ\nCjBsFqeAwhCEtk0mGS4JEmJsBLP8w/9pXDb6TKNVt8kUCV1mhIcKom7PD1lgmpLXiJQkvJZJhU2M\nLX3fWhb3aSolFCcYiwhUj966/FmNISmkGe6prXcffzq8tSjl8qvX/vrRH//gYQcQq0QNWu6TQHQS\nS2rKtgcA7pxjUAM/w73NpWYeKQMf3mMnAm1pBiCkba7rekNtk/vp/JxJRNubRBHh7p4ITg6oZaH7\njFSKMNaDbqEheerHN4qtqHYXltkJ8kng9Xjn0Vj86JufPTzulh/s39uMU9kewomcFx1g7rnLOp8N\nkQiBwhhJ2JaCOhJdCMYsjIpbHz0eANJqzI5oTRSq1mtbhUnfvt6gmX95GV1U3VG1S0KOI6G5GaOQ\nqhCadodLFxOI81IWZZp2zHZYplJjwNEzLFK92p6dL+98/fVbn50Wfbe+J5tJDdEN/SJB1afOrBMR\nEZc5WYBAKD2RSiCqEDGbLRLRf/fPlgK6CjA7lwEiBrFAaw2n68vJKeKCcig+JHgFzJIC+6mEiKaG\nNrcJIxhl1LSnEbJwz7sRskXSbawOUrO/43qdNhHv1b8aunp99HJzUdZ3d3uPbug7FZEyLvKgaPNZ\nY20yIZFmyqBgaBUilNFWTSHShycqYm1J7ewqHiJKORxyhApj2uwYbaYz7aTvEaGhSKoaXmoErUu8\n+QcSebHI8HDhcLsv9Cl6QsYOlv1kmx5e3+YrpO2XFx++F8MuD4dwvv5kpGvuegPFr6c+J0gSoc6L\nQ9Ecu9re6TlHNsVtMADGWx/InCWJZsYP0CzJ+cuLTRWhl+1Uw2uIlMtdWpqPE5EUKUWMh4BIWhxu\nCIQIRyzT8fUukI6OdxeT9IxRlj6ulu6rvJVHr57/DYSky1/88PtP16++DEcwJnf0C+uM8HqNIZkk\nBWb1UNs2K5yNt0WEVc1buKciJI5/51IP3s8cjIA4xFRMDt9shtOVREyHUr2MkTnW1KcShWHUgwpQ\n9+6BbpUwTxwhgppslSnL28OrlwVmCvT9YLuljffizWZ69PxHn1zQd+cv1z/8j1TLpHUkR0hKyz4r\nYn/R9YvUXKGS3pwRKFtyj1nkHD4zXkEB0ndWvHr95mK7n0hVFdOUUt9l3T55TVP4fh+opYqMeyxz\n1CkoEN+mtqi6MqCNeN5U4zaltqf53COMnSvyYhNJD7tOd2U40u2LZ/Kdjz4rL+x3v/o6Rx0YqMza\nW28U8HJzpx8UwpBQJaTtvpvnzE02FBqh8/ADFAl79PXli2nQoRu6bNbMiXNWp1C6JDFdbd1rQfKd\nDz0LQ1Kq3G0TA2CVjjdsyNn2lxeiiWAAZO5ZptVwvdNcorvYH6ve3Y37688Hs4e7X3xdh4JJokbq\num7RqQDlNftVL82suy3fFhHXmLnspAroUAuR0Jk1juPVz157J/3QdV2vnahaTpnT5Hm1ENb91Tht\ntlhgt+8WXXFGGiLK9pCC4lcYModFlPAm6xFVu4L1xYZaRNLq0FX6IZmPan45HN+u8uFzvzv+ejqW\nqb7SOqYAScurxbpTBl4/O14tGiNEBN5AhkZADGqISbSFZsGY4ReKSrc8r9jHYS8Ly9Z1lpdD7kqM\nMpwO6uP51X6zHRc27XQ11Apa0orD3hPBMgpF8xpos3cKQJ2ymFiMoPYx9Q4N9GOxheP4yS1e3/3j\nH10dHcuw/eLVIZdIo1JSTv1ykRA+fTmdHHcQMISiodIYn22teLtabMOatnPsxlzxTo6uutcYhZZS\nzsenEC6WJQ9Zp6uXb84vp5zKgas1C+EpO30qnpK7kyHaLdNv3F1UdP9msMq0R8hRvmYiGaYoghxP\nx08edZuz/+jnQ7/47EnZDbnk6gFR6/tFIsy/en56sjZtHnCMtjdbCRChICXQ8uR8SsJIMIy3z57m\n3muUOhYGwo5v31k/vOtv1kvxw+XrV2+urS8TlkdpU0Uh4l4mSuppqiFqfbZme08IVPcVm12CmyzS\nTlCqdSMCK59SX1e7x4sjwXevX13ferPvOEoqQqr0w6JnaLz6Uu+dDNL43m3rA9TbwoGgzrwF+Iyu\nWCM9AMTq/efSMTxV4TiWgovz7W89PN3sFh3q/vz5600stNTVqg+yWjKg7kb3tF5JDJOo9tY1nV8b\nDZd+cdhrpMHGi6QxduJwAdyLLcpq96vxdLUnbuX1m1SKFpJi61U/ZFG9/nT/8N5Rs02djXCl0dhB\nIbypgcGIZnzAmElMhH7wl1BlqIkmOopPp+/eSRe+SJguv35yEUPPUfPA/RRE18lUisDS21pl5RWK\nYdnQJIEIPHp0k2g/7DyM5qkfO6jvoWm3XOZ6+Hz48FY3HYa7z8KVDs/ojvJiSLDrX14cPThLKqF6\no8mmQBzaRk+ItuaYpDdvZLpQQgl9eHzZmRpm82qyyjLhkBaYLj/72YvapxqyXtl2V0Wk64tHFy7J\nVLtVYkhaLEqDZ9vF0jELohRH4xrDl3RLBbmflkyL/un04NHZy8vn5oToBLVlzkMWbH7++Pa7DzLB\ntuOriRuphAUaxa8RMxRkVbXwG9IiKcf3Xg4QWGjyoclZdlM6Wtq4/eWfPSld1smW67S/LkS/6KQ4\nUkhWR79YdUpoymh+RyJU3WvZw6I6BJIh3FuUw+rMpJweX2vCbb16+bOrt169AMxIJFus0jpbvP7R\nk9O37g2YFUUN69Hmed6gDROgaa1AJ7y432jQg937OhUKTC13eXF8fKJbt7P17slP//zrybIeZHWU\ny6E4LA95Koypdiltbi16C7JWJLTeU4UBSOzDjJB8EAYxnOpBFw7xfOx1XcrS39l/9qTvJgoiJKV1\nfzT4m2+e1kePThY3jENCZwY7oNH2cnjjlxOBUJQQVwlv26cgby+mKokCEwvrBEdJe79+/MXnT6e8\n6EIWiwWnXYV0q4HFEaOlKW2PFbt+qobQFJz5taKyQ3HJpm7CvLgyTUcbpuHoXAu4v1c3Q3/+4Ysv\n7+StWgmNGBbdtLm4qCf3T1d9206GEFqTC8Nws/SAbQENIsxVAYlwCb1ZPHN69k2TW1Mlm6lnYWyf\n/OSrqzENHbE+XeKw39c0dJ1UyLSrnSJtNl225UTTZEAEm++rDs980lY80iWbadEIWz04lOtH2zy9\n+6vdUvWjl+dCCemA7Of1ADs9O1p2Q2pU3Vn2CWuMHQ2GosIag0jAkKAK4gbpCpCLd74qShMjojPT\nItevt8//6ivvkBS+OF7rdjOypmGRAfh+yphKqle26kuh0xbbG2qaii4u925RTQxJ8iQSpUxMx/3V\ny2r3nr04vfdqWpyffvhxVQmD41B0OFktV4tFTqqCUASaGZQyoAiFUECN5hEbymZeDwEiXESpoKS3\nrEgnTAh2OUni60/8ybPK1MEnWw7cX2/rpOv1kFBiLFKkjCmN14n7qSai68QbpQBEniZBL2NKkkWW\nF4GFF5bDYrk/zu893b+8s5sm/enJwy9SjuJIq9WwWHTdIueGwwQDRog32HA+FkqR2gA7BSNEEFVB\nMEJmdty9o3NzCVOFdkbF/mvfyEJSqdX6wQ67Ec7hqFMtrIVOuKWU6s4PkYa87G2ekAikXlRbHFwg\nDMXudh6nflfgG33w+fX06K3P3pydfP36rZPXBUXdswyrk8ViocmS3tDVRWEgRKWpCWdqqLGRiyWa\n/bNFYaNYqRBUntx50bdE0JtqFbCEdF2KKDL0tj3sqclWQ5ejTLvdGKxBDdbdZp+OT84enlqjBkNU\nXnyxLyEWqqieajmOOE5Qsef33kuX063V+eXtxZdP1Z+KuvUpLU+O1qu+0+ZVLZy9j4Mq4XGz7EAE\nAtVmQiZNDdm4EhFtzzI9+gf0EAElZ6UI1CQtVoOqdENfttdjdVsdL3ot034/VQaR1H2qhYtVzqu7\nA9oIKZzd/dXoPSnJYWoXK8lnUOvyLj5Iz55gbZev3/XXVycPjmMqko9P14tOYQ2lDKeIiYdTGpz8\n10cdImrajAE4qyc4Q4Ntz9vbfY0oXkWlbZOWPBwNKtKtOhZHUulOjgaZpnG3m1wgwy1FSBQsJMDV\n0DiQJGlHOaIgdQbr+mWtx0MUy/3kX5/eLtcn+bTuL9bjXu9/e1VdFsd9VghEdV4MJkJl02mhCRsF\nArowtO0HgaoKVFijQcMes7/7vZMaLjUQwZRM0PWrRWaNZGU75UW2/uhklTgddpv9oda8vH2cGMI4\nOANztmq3//L5Pu/KMoQmOSNfrs+mXe5Yp8uX37rQ49WbgZNvRco7f/jjw+kqq2lKkDbVFZ2VhxBh\nqDc9jMisKVTOBlkGCQXobYUFyRBQjh9dwkcdwtlnp1jKuY61ooxjZBPI8njZRYzXl1eFKun4xDQ0\nvO4PZEQpCnFRgMpDiLMLj958752cvLV39cCufr78oMo+DhcY9KrsPh3fPz7qskLoKrPejREiomxa\nYffmp0ifrXh1tqk2BajSrCeBZrlP5kf3Ogl6LbnLybqUF1l8mup+M0qXTHPKiaz7y51Pjv5o3ff/\nX6lU7Q3We5CWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<PIL.PngImagePlugin.PngImageFile image mode=L size=200x200 at 0x7F1488054EB8>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from PIL import Image\n",
    "from torchvision.transforms import ToTensor, ToPILImage\n",
    "to_tensor = ToTensor() # img -> tensor\n",
    "to_pil = ToPILImage()\n",
    "lena = Image.open('imgs/lena.png')\n",
    "lena"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMYAAADGCAAAAACs8KCBAABTfklEQVR4nE39+bMux5EliJ3jHpn5\nLXd/Gx52giCrqmvpru5qtZnaTBrplzH9yzJpTCYbq+lu9UzXzmJxBUgAD2+767dkZrgf/RB5wYKB\nNBK4735LRoS7ny14JQ9LC0IEIRcsmGOgY8KSAGWVBsGyVIfkCIpCqaQIADKlAWCSIaNAtX+l9DLJ\ngSQRniYREJicvLhMBs1GAmACRHrCgxa1O5uTtPl7bJzpSsKqJ0wwJYxhCGR+eFKERGWKAquZARQi\nCATpIQGWAtIIMpAGJgiDmCSVJkABQoBAmQTRqgAiaaqWCRISSQGUZIIskSaIcCBMcCZFCkpAxDSb\nABs29ysjg1i+a0JBSgDFXNvurUEiABOQJCEIgpkEIAiaiQAhUSSg9nVKECDRIJBmgACoSpTApIhM\nCstfEqQQYSlYAnDNmUKCBMyMkhlEwQFIJXciFHbRz8sXETCBEgmCpORDl999ZxBlBhJoTxskEgGA\nUhJIQaCBkmQkKdEAQggiRaak5QOaQSQkEKAByBSdAGlGThSN7bVKBiQIEiAY2nswSQBEnyaDpOE8\nJ0kgTYlMMYEkAmH9ZnwfJ4b2vUuAuZGAgianUU4zkjBIApKAEFJbNwBBZds1oNKdIFIg2uKBkYAB\nShPbxwwKlAoUTmY6IZqTAM2BtioAgSw8yKiqk9UsQCnBQCLYFilsKLsbWxVDikgISQMS2X6+KjOZ\nSCUoQIAMRpgtK49gspAmmkmw9hXRaEDAALX11xYzAEHiRoScKbrBFEFoORQyBZgJSJMSgteRREa5\n4igYQAoGWNsYVtbl/s2UJqMRNIBGQQQFwbpioGf7QwAJI5mJTILk8kUrUim1FQSBJgUAc5AAgfbt\n0ROiO6UEc1noVZQyQbKdPgQlOkmY0WRuuzQiYrWdQsn2hIH2rOm9311z23k7hQSwfe0pSEYtu5od\nEkhRREoQLKOtZWV7wAQgMZUwKehgQlS0s6I9+xTArAkgITOkUEDSEbC2AtppaQoJSIlCsosjIIVd\nWAWAnKGEVIqBHPz62obCoImPK3GpAQJgSrWTVxAdaQiIRND98WjJbC9vJiZBKklVpDIEggAViXZu\nsa0EWWYCxkyE2v9o54VBAkAlBYhKACz7SiprfznOSToFQBkR8L7cjH1vNWXGtmQfT86lOEWkaoJJ\nmqLtBFK0tn5SSJkbIWUkAQPN2Y4YmkkJSJHtD7I9PdJI0qXlT6doQpAQRQgJkLLlOAVQcDSjUuf9\nKKUVa2s0wa67fYcSkYIVk9hOQwrMVgSCZmA7ogFrpQtJEJBICKQkUt7qRPvIIgQKJQDJZQiXCWQr\nW0oREClEtt1Fon03okC1Cqb2SoCsHNYdlNFdTeMalYCENLAvb++MKjCgWtudgEHLgQKADsilYPvG\nCRkIQ+bjT+VjXSMFM1EKwEQCWSWapZImLRulVQd/rKSgkMo0ptqvan0PE1A7G0kYPO/mjIw8fxpB\nAu00Yt/vdmVTJFLxWBJ++Kt9DZnteGsbjimIQCaJxxPPIJkBomUrk+YpgW3ruBKt1ptAtGN4OQkJ\nyRwKwkrm8srWjh+11dseNUH2tXbFUHWGY0Bt4duqu32LLq04JSsGupHtRSQkJQlGRaCVELYSDhFK\nW2ouJCCjImsYzajMbB0hDWZSW0CtgbG2X0UFXEoiJe+IrGrNkTIhpSC2/dkKocRiD0Eo63A5jm3Z\nwVa8vraOFq3epVGpEFvjodYc0Vp30TqXXNY0CSBlgiRJJL2Q1j44jAkCTEk1AyTBbB83gVxWYGZr\nK6UEYLTljCJpbXMlUmYwLdve56MRSl1sJonItM7vbm0oYmfmKUSBJc0sBSDJTBBKyYJZJHCpNQmV\nBK0Vlla/mDK0PnbZMYagpeYk4DC6wROWMKVZyJPSY3MqkpBa644k1P4GlViWH0Qr+5VJwXKpsUek\nD35/Z4VwRMgEoKRBcKWIlMGSBC0htD4FMkowtX/42OVyORUC3n4KKpAqlAgOJaNmTXRd285YziKJ\npqXcJ5FpubSJ7Sx7PDTShaSSINgf9yeUgier4ynNV+Xhhh0NpAnwGaV4gsjHmru8UVl7GRIWWlZ4\nGIFUygCCzPbCbK03EagyrDL7fkjL4or9/SFK1xmYBNs3sbxCe4eUESbJRJFiEkC2xy1rHSb9sHKT\n0p8xOnbd7S06D7b+NNMyiyAaUxBkXIq3qGX4QLZJS8uPuCCIzHZSp2GmC5ZZvWzhnHLweV+U05C+\nvdSL/jiWlVk4gR+GTCbpsXz7aQLSW9NMJgwABYYlRLHU3bmEzPXZ/mTd3d9qQACWnmSVGUtwqcqP\nRQetCGLZ4wJlIGQikEFv/wagDCVlSiSTKzPWsUTpQoQjD7N1h25zPu1uble9SYxlAcEq2/MR0lut\n4b+qgW1ZaPkvE306rqQMu+q7bn+D3sVwZRWBQmUhHSmqDQ6thFiAaQRFpLHtaq9trkjY8qOEpLQU\nHRj8UCyDFlPII10oYMRYuvPLZ/O8cbTHIGvT1+wEYbTlKyTaxLRsIgq0bF+rfL5fWWbG8PT+4U6l\nW1o5oHqfAGz5TypZ2DYuBCQyBIQIqTozA9nKYyYQtW3seRpRtlsP91osse6gKTPniqQzjMjjbrz8\nfHW3A82ZKklCWSkgrU167cG0yax9le3o5jIaspvvizFrPutfRbFIGjLNaY7IQKFbJBUV3sb9aIMC\nYV1YO5QothkYlORtFCazyl7cb+dpqsCBmNUjaIye5Jg9oxjDUuNx+8nzuawtTUi0WYqSJCgdbShg\nyihTGNs5L7YD2EDv9hsHhIvvDiekVTBpqnImullGZKvH/niI0Ag4kXMNKVs1UXtoJOyx7qraGfe6\nuz2iFBVjGA7Bku5Vss6i86w1ldLxcPXj4e6w9OQNImkd4+PEgoYQiNa6O4LLCGwg0OseRt/Ub1bF\nlG4yQyQ9M2d15gYYYwprhUCt2VcmYcuBBToUDc6BLbADoj7Xw/44SSIKJRYVZpVNk8CoXYFQTCnm\n8R4fPpseRhX8q1ERyFx+HRVapp1loFQ7aRq6wv646/vN6p9utimjM8lZrfPMQqNSQiTaljGTtLx5\nGEQmBUT77Sm1qpECslu/phUvxaPmjGRNT4BVXSl005QOC5Ag9XB48nl/d6whtPeHBEGHL7hDeyhJ\n+2FwbjgJhSRLuYuTk9/+6rIzJJXOEJy1yoYyFiUspwM6BEgGjEB6CrBEAbJVPzqtyqi5LQTByq/6\njgxA8MIk2c2p2c0szTIRXkV6JkWy7jdfXD6sV62LRlv0AhKPva0sANEWYOsPP5cAB3n/i1+dnrSx\nyBDRg5JHGQ4PRaJQyZJlAeRAGJiUGtykpGRA66hgCcmU85n2g2fIs5RgZYlS52KDMBdGAuyFHil4\nO25i33/85O2m37gEJMwSUtECPjAbQuPMx8+RyxMBkXZS/+nb7RaiMQyoKHOm2K+nh10RoHFyyBd8\nEu3QbtCG0sQ26lBpEr19c4S/+G1hTXQiw6J2krBCpayEId0T9CQkWprAKXT60U/17ng+MGFKpmhy\nS9FCloh2OFEwMtX2TBs5UTa3b1YrKouRqqkuIML6w+7+oojQ6N2o1l64kOZKEkQWnwVZUpA5ISjY\nOrlcvx+HShSrmW5TbyFR4VKpAHpY0mZLklmoWsENfX6+/cTvLwozaTJRppAHZGQ0QBhW2TDRttUZ\nBrNNfT+vCZgFJVkHJKSO94f1C4NibhW/AQTtQESaUpYzZG26pBQQ3cwJpDq/LwDMMrxYFGZINS1i\nCJQigdY+s3tXx+O8OjvpMO/ev+Znd7tQmzuTfNwbknnOc4VkrkyS2cY5GIVVfjetTTSnSKZ5zgj1\n65hWL7tiyqkhw1pwLEhKNBQGpaYsSYGy5CMA7SzTbLQChVv1ZKWBTANjhmiJUj1ZqvN44HoonAQI\nGm+ffvB+HpgWRtayHPMywMuc4R0WQiL9h7WNoXu36xEQLKmUlaiQet/fz09O54IYo/c2vi71O0F4\nESVZUCgVC0puEEwyijjSQCUK0sEsZF1qKc08PIlk8dxXnqy8ThCTNhun+kf99WUntMNcSTM99oFH\nP2mFRMv06JUQu9XuDYtAB9Oylk4V7KvZrZ6dzqPlOLtRQVo7a0GDvGFpUC4zFEqbcttMq8LRKDKy\nRC1Zo+HDc02WdFKiJiaxvx1Pn51gHGubiPqsx1v9Ke+j9XZGgIwFXfPCnEIwWmZDLxKCbNDr6Enr\nmGlKDpwDTseR51vNc4nZihKeEINsO0MCsgAeC+7cVhtBo5LpPMoIpXvAai0mHktXaaJonOEzO1l5\nkydnjClYEi4LpVvqcPqjfz6uGzyRvgDyIKXO57m3hmKICZIm0+Cv9s96r84Kj1CnGUWcqbo+TWgq\nc3YlxQVOFgWSuXR/ssyG5zPa9ImgpduU5hSMASjIEtG7yZIok8KUxWi3+80zTTW5QiqDaR2TQOw+\nevruSU9ZLjCwQCJJaUaaBHpAnpCkLOv79/00BTPBjPS+RodQJw1bVBuHMrX2o8F1JFwQTV5hgLyB\nTLlAbqyQCA86oTADAumelWLW7Amv8yAHsdpd25NtTIiOmZkWZaiogbQ6dz+6uFttHQjHAhgBkIJi\nHdpzWI5N0IZ8V4pUjEnUsK6OJnUafdWXjAmDoR2bNIRIIVPIRApVYpJoE5VBNBm9iF2Vl0gSCu97\nS6OQBV4zInt2CK5vXq0/XM9j9VXWOiWHIad5nOi9o96V//AX9WZcEOsFnjDSgDLvYplsFnC6794/\n9C4rlBTpvc1yI1S59hmRG5r5MmU/Np5Gkg2iWo5zNMRFCyQWfY0Gg6dgzAz3qLCkilnFHCHz9/cf\nfTDtJ/cUUEpXj5Nb1/VGwSxv75/8u+1DYOGWkKCzUa0519aENTqTPhzfuHOpJYSXSV0K89SdWChy\nZSpOpCUI2NIHNHrSKmRitAKUBBvblNOaO5PkKUBKqPOZvSNpmQaHe31If9kfSZtkcquKvkwqIbDS\nVLt6c3z+48PtmcGS1lZV5UIeVrMFUJIBg32fmyIYo/ZRO5urGSNnP1/NmdGVkGWCcJCGgBRSksZW\n9CiKUgaJZCaEOn1Yj4SiEWqZxqgGMmbRi0HS4UEvns4jhHCYTXOUjlPBDCXopLGU+ubq85sp29hH\nEGlwo+iY51AD+iF0/fvd2iFYlE4BYoInsmK7Vsxz6SQWo5SGbLjagotJluqQomXjmhpfm7K53+zo\nSFMag2YsE4lSJzYaJjL2JyfDHqGgZ5nFHkhYdrsVYFJKsxNx/9H5cWsS7JEEzgRgItw1hcI699X0\nhi6SiUygt2ApOY95dqZa66og0owii7UZXBRtGR5TUBgToBnhrc+xenw+HwqSoQRL54owenc8ukOO\npHb7J6d5HFhH6zVN/bCqURMedUhyjDB3EqnD6s8fDvDGsjcOlG6Cu3bv34yb5590h4L5u+ydMEfC\nYMjqljEd+1NOYx16CFRhEiHmv9oaAOg0sFjjSNPcQgSZs334Sr3SmEUzIJLJKnelFVWb7tcX6xo5\nqljcoestwqu16Voh0Q2qaYp6//LpG6TLZgCQJRTpIPOt/dEf+zhf4XT+5fu1WxirrKu0SBjmxMlT\nn+daSivWlsiAg8hseDweu2OkZlGRlRlTNppzf+HvoCTQSwaFDF5U+lqtTLs67s9fxtEx7m2Fg2+6\nOotRiKou5zBbe82mNWCM9x8dJ5CiNRmGnDDG+L78p5/Gt7+/Ce7Ptw/GlBFFY4LGrqhOvBzqFNY3\nIIS2nNmPgPZSzJGVMi20MUgnIoGoH95FJ9IzwOJGJmSl3qsvc3AfzzfzEHX0Vb2dzlYx0VMppcdY\n6cbJPE3mCUQen2xvIpFoJFNrpOLhzcX/9MHvv47z056YvmClikVagZBpCCnXa83B3rCIQ9SYILP2\nfs0AWRuaso16bmy4npvmcrofCgCEAclZbhbj4TCsMROH9ZPBOO72fZeTlVrpglhEjLOteiQxmWZE\nZBXiaH+8P5A0qA0WJtXbmy//8/CzV6dnnnOdpyfneykru3GWeyQVc27OMc1RLGvCGmiNJNr71II1\nApl0hhZsdTnAwDKdluMhBNTSKVnRu6Ycde5jxP725LkCh3DqiPPhOLmDYVHKVL3vcoJV2EzI2Q3F\ncn746PmNAr4wQow6vj3+h7/c/f3Ny3VMVbUecTU2lMTMMNFoDJz18xyltJYesoVFtkYH8ZGSgaXo\nTEJsz0bmCdXzYzAJeH88Wi0rZU2tzqcH51jPz1TqfvYNaseplkKUmiiHHbeDZuAYRVa8c2fWmqwP\n0xfHXUNXSEl5/3r1n7/4+h/8E58rEHPW2M5QNr63ptExz9tN1rQCZbZJqYCwlHUzmZ4Lyc9Ee86P\nnKsxJWIcTycWO0nPB63NEaFksUlb3teLs303V3i3LwOO8zDME9FzPpS1Qj5HQe08bZjmVEcQyrvT\nHx9WpyTAtNi/uf/yL85++ebyeZ2l2dUbgCqnzXM6WMSYJju3Q6hjNpmIgKZBohq9xKaCaRguYTAa\n5U0OQVK+ycNcUjnztAOImj2Ntsb747NuWt0f7Czvy5Bj1yvM5JgP/VpTMAuCZeYQb+/NS0OMNO8/\n+iKO7WzUw+vyf/3L+396//wiZmU63WSaSyNpHSF3nw9xMUxTuLeSSVmqtCH8kTiVIGc0fRkE5aI0\noSXJWK2OI2wq2ZXAhOHQW3aYon+Ip+vYm6G747buu2K24wDDPjcKsxkwwqs7dnFRQkhHJuMYJy+/\nHyyp8TA+/eLstzt75mOrZI3R2V8rk4WVNEDz4exl1jRvUhMIMFkbS1uplgRrHYEDQMAcqQCggJTH\n0u2qO+AetWKl3tmxojzgg+1YNXb9O26FgZjHHlHKLk4HzOp6MhTs+nzFp6ygnKQDOd6fDBWI8Xj+\nZz+6+c1df85pUcEYJcNNb64IKULKae7OOQeLR2MUDBJKQ7TQ5nAaYGhiGQCuAE0NTRCh8bTedf3c\nmY4WPuBY3OZjj6lsY8pjf3qti9yXLiKjDKtpn+syz/00R69pKDTfHS76FJWUAfTZNQ0nN13ddx+e\nHl5XO+uySpKZzAR202FLuSAmhMxydhk1gGSrdSKTpamUUqTMFzLOxAqoJDzSZIvsQ5ovJ89Mg/ra\nKacyxIEb25XzuUYMdoNTAM5IrVI57y/6g1CtDkf3WsZDl1sfM83hWU3mFpyOq3rXPznT+yP6DafQ\nwrEqZRwe6tADljWtKF3dlcaKYo3UdYskVQjLBeVUG+OdVtm0BUFTg2Ak0Y44O7DIIp3OUGGOZeA9\nthW+Xw3vy0nOK90NLKlhV3mJ0RkBVx8rzbv1ANx3vUWphOABi57j9nR6aTeTdcWzgqQa2lBArd/Q\nQRgTHkDmpq+RZYGXoSacLa09fyRfTKZkIxs1w5LGVOs4kHO/vieYWThnbw6lr/IOT2aqDnld1jNP\n9uij9GV3rP2mRvUCqyrYxoRThvk2ASVHLmrNmXV/6nWXxYacGxXR9BluobJ64wo1yQFc0W8Zaf6D\nSKDxwUW5KASUZWEZuMjNmhTzEd2xsJDPXbLrORlFzZ2bjvFEdsjTuBv6uYuD6so0Bg5rR82+jh0t\na3eIzoIx9/TZbQbmzjMN8zhsyvy+L957zgtyL5nv4QCLXocjFQFjmcduW+aqRUK3vNv2Tp2QMh9p\ngKb0bZKkxsNbUrQg0rPWKazSRehYPfCQl5HVtrrebOVedxp85fZuvNzkHbsYSlhxTVmcEYUhFlXz\nMMkZ98fthV/vV0PpME/pbAg6+3GscmLzcL/NMFsXo2E6divM4baQbA01INwcwUf2knoEIQkaHUha\nSRiaFCKLCRuzYrOYXPWlXvNSjBzizbo70udpVaLucr/ZaGaZ4VH8cChE4bQjOWQTNnFdOLPu/OXp\n7r5fD24ZCSeaaLDEPqKL2U5+iwFEjhWaZpSt1WThwrNBjd2LkgvhA4Bo4nITlWoKGqGxumxC7Ail\nT1125TittarzcImcy8ndw9kqLYMnpR5O8mG7rccsJSfrjurq2NleZUjM/TCKFrWHjXmHi+3xoQzF\nQjkbrEEXQCm3tTvbvz8v9nVZYWRJMaMb7KpO8iQQ3pQrbWJvwqempzG2XmsRdigFNtCkIUmILsjs\nuuRqVncs0225mg/w1W534pWYUfiAkzFPOM3eVZahHoaCVb8/sHeUiNm7SBBhcRxXz/3dQz8URp3E\nRUgqEOX4UK7wfVfO3t5uxrrGXIvJ6KeogjXZUNsXTb5cPJoOS66g4QcSHFnaOcxKUmAg6zqq84ik\nxoKtjrwIzd364bjazBVzX+fpxMcs/bxHty4jVDSt57uuhyfDRE30RJm0m4cLvVPvndWI9iEAJAUU\nuxkux+9PLuLy54NTs9JLpMVqqFVuCRmTskcojaXJhJh4lPOqSbcaPW4JmKVKioxcj3AcVzGt+y7i\nuDrFlKXc19NhLDEPdR7W8aAzHdllHpmD0epdYhuVmvr0VI2+H/s48HSzG73vmGMwzRY8hyBsuKuX\n880Lv3lu35x2Ujgzg/J1TmFMUNHG7Sa4JIpM4iINe5QHAPBZsjSxjexVdEnrLLTCfjamjt32oUsf\nbqL3cDkPdd3bg61n76d+9WBTxWBz2DbnTJnPRSXnPoQHrS8Pd7Vf5xyR7H9YJhJo3XR9rvnl/lX3\n8veHZ70lY7AqaNOPCTdJYrqaygAwoSQtw8xC4Y3xFMCmNzelA1FcTCYUBlI95sAw7nFOO/bl1sp2\nqn3YUf2m3uZJzwjMWpEjDm/9390j0/qUWfZWxix7y+e8PWx7YkqYLx6VR2qJ5e1mU/2b683np//b\nypsM0mdO/SbnNGsUhSmtsTmSWKA01x8EvUy4MpvxgSWbwQDGH3CTqe+KupnbOrOsb2MYIKc9rLoY\nZ9sqju8KI/xyrTN++s0N+443x17bIfd3/vHDbt4+m6/LORFTmLelQS0rGt5Ndjq+2Xn/7PPdt9vM\nVDLHkJ2WacFiG0pKLVLYZMGj+Ksdta0qmjzTmQaipAwLMRp99rNluqS1j7D1/bTudTTU/Rpa39ja\n7fb9yWfzy6iH16ebevXyMA958dTx3T4vdh/8/dVn109XN/vVxmMGncjF5kKmSPQru1+/euicZx+e\n//906gTltBrd0zoD1vQGC0jQYByoIA3yUINGLNnUpa14N621N/ImZHXweQjVuurrA5Gr/XHV2ww7\n5EDkg62yjPH5k/v7bpptT2j05x8/f/PbExtwtx/tSffb//Nn+2/zop9C6U1F2nBPo4CyXh+vbw7a\nSCdnJ9/+46VTyGCpM0/LVOFsHEKmLZwaAEMhBYRhke9iEddYc0o0mZiSYpqX42UYvEYE7pQDD7vN\nwJHDngOMe25Yj7dn3c3dqtbT4fLZLs3e30EfPvt6O7pL392+vHz3/fnaRC5iKmui23TZeu3739yn\nWYk4vzjf/G/3mx4UetM8nT6POd2byIqkmKA1XAdFrmDzfZihaftgwlLGPWtjtGiJ7u2HTkNiLrfo\nFX5c9azmu9hIQVtB9n61edhfXew/Ke9zvD3hezucPD+Om8tesnH9b8+/f/fBuo65aPQaM9c46lO+\n/+4I62HRXTw/Hf7u3TM3gYoy5bjxMX+YiNAUOBCazKgwaYEu4a2mY9EhEaI1/TwsyKTUP/xfhik6\nx/SkL+MwH4ZSwjBml8Y49hTut+txd76p+O6QSaB/qs1Uj8qP1OHN7q8u/677wA7RSMtFRiGZYdh0\nb36LPEOfie3JcPnuv8e5d5mQQzkNm8golBqtzQbENz0fWQAIzD849ZqymGjr1ti05pDgqzeTAT7F\n9qpiqEcvpRr36GmKeU3oelgVH/x7zsgy83xr07SK04dNPfsN7p/82+G/rD6qD8kuRcnwyJtyWO3e\nfGdnymrrjZX+0v53vHAX1Hh4O72YAh0AMB+1LQBas0QjldZIh8ZTRtO5WMpJy2wnmJHOkm/PMoIF\nh+xs59YFMOYgWUxDX2utT/vY336H09OhRLxc29vvruPh/uk07R/enf37u7+7+GR6qHBhMQwSAmx7\nhV//Qzwbxj1Pn132rNv8X/Ws0ElEmgt+okxfvCmLLwmLvwVIS6AzKQ1mf7AzNKdbldonoaVR8tU3\nK1Nd+TwKx+w2Mp/GHpSiX2fxVRkOb1/zk/Pr+8r5/MRvbuaPlefvrrfAp3/5zS8++CTG9MLldZpB\nYPNs/c3/sXuyWZ2fv/zwqq/389Xqr++fGEoq3UAdj2uvEYVSQ9gWZKe56AAUZFOEI9N+kN2mpCxN\nQu6NfpKR2Nz834ed+lV3WF8ftwbwfjozeRzCr7U9RXf/tl6cX9+V0wPw4t30tv/08jfnv8VYvv7k\nz3/z7vnl7kCSAfljvSjn/bf/0H082sP2PGKck93Ls79++7SQSg+a4Vjj3Gq4h4imuWxqpUfBRvO/\ngZScULZpRKCxeQGoFDJZjBB63px1mYfutEqDGTNPDHVmnqzXvL0JflOe9m+n/gozXuTrffeiu/7k\n9vDFzeqzD3/27pP1OGWYiWAEKMn68/r/+eWHpx+vuknTsVZw9cnVX799OlAZIDMox6qLUAfZQkE3\n4XsrhE2bATVBXuSylNp0mBnN5pIioUga0a9/vlqX/ZH+8PqEY51vN51qYlp72VruvuFqs0qbnnb3\nGPxdxcnJ9/3bN3/1bv5s/cv9x+s60R3Z5HJpZoS++2/704+Gt6fux5qQTj7e/7/fP+lpBF0wIiq2\nmIPe9FGtjWygMtXEbGZGt5TgDXtuQ73acGKAkzW8WW9gJ/tvzrueh5OyXWuox8GUcA7d/ZubuYZP\nm+7+uC8fvE07i4d+KK8eyven2199sf+74Yt+nwlSXjJkTcpaE5vPz77fvA/TA6Ly/KNv/vr4tEhV\n7ohG2pe+TlGYpAyNr2iI8zLTtemPbGqyzMZ3ND9TE72IYGlbI1Pdyb8ct3HZz9fPDllkPTE451L7\n8/X5y03tN/e7I66wNw/Ap/udD4fLV3/y9B/7TzBGU48yQ056Tofw7upqHLfoDieO8NNPn/zsb7ZX\npZRiyswEnLK+r6NKg5qxCNeWpQUAZOHyf5Qli6WQ3vZItgpYCQrl0d67ff3bP1Zs7qb+bsi6FivG\nYXXPbb3BCpuTKVd1Gu6mk+dvvay2H87js4uL/fmrq4/rcS4ywDKrmVEBM/eaqz4PL25uVxeI05O3\nP5vOVz53bIsHoM2JNaoGi9Z5NFN1M4stvEib+MjlXeYPo8ujlSog5WL5BIVy+j1OQuXJQ4+9YPP+\n3txWV/W9huionQGr3cPZeheZx/f3N8c3fvv2jlfH8ZiZMWXWMLIesxR3JH2z5uGrFb9ll/Xb308X\na6hkysyQYMlAP8yzF7Y10fpxkOaPjW5rP9jAzUS00xhAmNTc7WZGEYbFOFJUvcPGai+tWOfhfGPT\nXbcaesSdKkpi+6CPpvedkTrMD3g6Hifc1ypXCIau0HLkUCQpItj5an79jNddT9+clZS8iAYFm91m\nZbVag28ICm6Ps0VzLTV7ezu5jFgE6AQcavYzCQv2DiKVIg1df3vj4km32/dbRrB7+G7YVOUYOVzN\nrMfz1THgLJg2Tx/qs4urs05DR6nIfR5rrNeNv2bGXJz9w3y+t8rNkc1/i5pMuklRyyoq+xJSg0AX\nyw8TtAZE6FGUIMXi2WsCxjDLxiNjAX2bXIHGYClv0CVsrKtVanK++X4a9mFDrLrRa6zePnx8rqhx\nuF9/+vrhs9Xos8wiMiMtJ9nKl8PcWrVl/76b6K5dX5yAzDsyK0uEViVr5zXNmxuteayg5sEQRBqZ\nzWLbxnprH3bRDyPbnkoYiZoA3HbD8H5cHYPTuF1Nh64bTof5xTQaoLtVHdc58Ffl8uVT5cPwp8eb\nH60f7vpqxWpAtrap9FREBFAMZvQOBsnHbT+Nqrnsx4Z+pHXrnNGj1rnSnHT7g/WZS2NiTVbfHgha\n9QNhtmyfpsR3AuNc0sRSvlsNN8PKrNYzq75/mF89rPqLQ6hYd5xOjt5NJ4HrX7zLO/3F9Xc/Wt0M\n037SNMrMBt1EIWHm1vzEhlJUwMPqtnRKQ1+cEhECHJGrIUfvREOt89xcu82CwR/mumYZRALZfDRk\n85AuMjCaeWNo8NFJJITNQ/aHUoY6raD6XrsYYe5HX3WWftxjxvZs//3TjCn/7PjbF8N1LT3TXen9\nql7bgFaCc9FvAJ2j42T7I1aDoQqiOQB6QZ3XmtQ1Y4kQkUnSmrN9WfGk/WA3fpxjl36dAs3LMrqL\n3I5vPSlu8pvTgnnmNo/l/OrDy9luR58PtpqUPSp12FxdvXlqu/GPhq9fbvakUSvrZND40K3UFHZQ\nG18RcFSLrP3o8LlvcygyoVTNocTsnVp3agQyaoi2mDbZOMpGuqY6LG+3oYiUl8U2Gu0Dz7dmRspW\n3+EF87ASjQ9x+yquUKpOh2tFoU7qfH739mL9HLefnf9qff5gpTgO2StM4zwUAGp4QGuJZDC4waM7\nDNMq5/YgGq2q0FlfwyEWt2z5I8qMOR+pC0BNqmFIcc7GfMuolgAQZgpJNBoSXBuM5rbGN2c+DwZf\neb3elU9x9Jie7OyslglFVnf4/uvv+2cffNVf3EPG+WAbHTMDA0tTo+jR0iCA7ORuyl2uWQWZIiST\nvM5ca1ZpeTDmNFtkz3Vqnnql3Ni8w2aANfPiYtNVNjj3MVeEHpOpZsRUnn7zNvs1u6j791rvxrvN\n2r+ZeDEGp2Ee+tdxVbpffvJvvpuejZOYGCr6/aQQRTXNMpp9u03j5uhcoh+GsZMAeskZxTXj0uco\n3nxCLUFEZpJ1RNRZMDYDnhl+GDasTYeP1tpceq+GN6YbIue5O746Buc6wWN6EVH6bux195E9wObT\n/XhZhsFxjd/dfHicVcXqF8Oo0jfpT1MvWgssyIQU3oFMUx43/Swo0gzKUAZXmMKZTSOfaFIjaxua\nyJjJ1kelpMhmCGy5CQ13UdoCKaZBskx559LpxU/+8o/GCpbubjg/ys5ua33+qe5R5P0BJ7vzXwF/\nHG8+iegNRNbzU4tu1XW9s8lrGqfS/ia966tojEPnpaydyKjeuVf1njULWn4Il16DwB+sMlElltZR\nwUyI5o5q/1rNs0MiDAlTMzkeLq/w3d2wLSGu3x0/3O6G7n0MiFcxW4lyfhwmH7j78PP/8cnp7UYz\nxG6wX39/cbLq3RmZTR/iiQbYF3N22Dsl33tfcxS8n6CgEtsyZvFFH90QW7agmUUbJTBRS8qAlkvU\neFws+649/iaASQphFBSf3fz9/nCtn+KXc3198fJ7bv/9r13zXNfWd7Eua80nw+vus386+eydr+6j\np6+mX+/OOHNeb04Hq0dXbdkXSVoZPA+7h0gAffXtnN2eHhUGZqAoYrW8pQaEyBg0NoEtovVgxR6b\n2sW70fzOZEAJRhFk8EyDJVLda7PLOPnwcHOsm9VJ3pSzu36eTrkqfWZMH15p3989/Mld/OQaax3Q\nd8PD96dPP8sPDzd36SeXp11tcyhl6X2Z3r0/DugtDfXwf1v189k6OGUJeKq81MzCfLRx4bEzhGAt\nbQpMqkjGlgojaVlHi5eEhDX5q8GImeGbePNsPNj+cPz26WraXv7m+KP67Yv0/prD6ffyfPWSr7rr\nz3702x/N89Dtjqdnm9dvTk5sZy8evrjfHe7z8in/VQTB7s3dYOun8fLmkEmLHI5p0VukAYZix9pb\n/iuGnhRcaXzcKwnKCmiiFningT6tyarmgHG2BlQm5uRJP3r/R3n6u4+eXE33/iH77vNfdv9uPb+Y\n59MX382+Dlz23w2f/OyZHvquHoer899cX/iEh+u/ulc+v9BBh0FSMagqjw+rz0+uSt5iXh1ZqqKL\nydpRr5x9q0BBQI1DbeLahu6I7R91GbJCpWC5RE7J0pIyLPBLLlZ4sFZbl5xUjlt7+vZXeTjrL578\n7Lwcan+32fXs5sPzr6d+NWeZ//TgZ2+t98mvLv75emV1svnwTze+7ddmVbNnOIO+wvEkrt+O3nHo\nasQGhHfqM4OohiNfRoX9UCqzfeVS456FFgxky9yktCW1pFl42cAeIkhrQSnTzj74ZEJFwXd/+5v7\n25tnz0+eHHujl/WDOarVb55+kbvs/d3zk/ef3luxqZ5d/P2rVd1Vg/eHru6+iqEYEXCKdrLef/f1\n7+6q9UNJTA7E3MUqq5CAF87savRL6lYmrLUjNLWC07pBKmE1M1M12g80gFAAsmYCEalMKI6bn17e\n7hP9VKU3I7/8/Pvy/vcX0xmiKLZD52X81dVnl3U62pe/ehoHkbk++9nNJcLh8uFP/3R93P/iXd9q\nkuCb7W//ZRzOrr7cHmuxoxktsx/DbOFZw7YKdIuJoeUvoAltnURSrebBGlzU0C821gC0tkHMzJ2N\nYIZtTqbvIMfNpjzd8/nVP9eT3+1PVvxuwDFu66aW9fGfv16f6/jhOD+9H2AaTn9xc47K8xwehrF/\n8edPuxf7CIOEsGH7zW/6Up9ePH9hU9Kp9DmojiBpxmlabTPNQ2p5JGQDoUD+q6lDUITJjAZCjwUW\nENF8r1wMqVZWeP3w1QcX++yeb9/fx8vnP4uLm36wB9Y+a8Hdc1asXDe3+eTp73/8MKrrePrN7UWd\nolwOwzO+vb7uftqdlH2EMuEcHn53cXravfnN2K8sZ6jzLnPI0ocCCsEGn8K0rKJlrFv6yYS0jLCo\n82zLKQaFlrm2HdHujd5AYTlbHe5e3fzy5sUXT775xd+M8yW/s6t9rruVn+ZsXecPRR3gXeJu/ZXt\n34jFNt+9fnZ+cZUPx/Pvr7/7+Ol4iO3dMHUhOlnspv8Svrm71ASEQgeVjD6mksnCHjmtvMrbO+bj\niYuW07KII4Fax1o21mwHLZWp5TEkmt9PxkV0cfjdLZ/3f/b8o5++PnZTrp8cVp2G/qyf76YeNdHH\nN9alhJw+uJo+35+dDf3q9ZtzaLM54UP55OInfxE2f7dxDijMQOnzdl+f7A4f/ZtGDhe4wQ5mtUdK\nAVUNNcwXMS3EIrT0O5CmzARirui3QyltRKFRhkXMtJzRbBLKbvUPPP504+Xs6jd/e30xjx98qbe3\n+HA33FsHuxjKrOxqn6UD6+HDrz6+CKCsfv/mfBWz8+pyjJOr8p6hTZA2m7IQiOH8zeUnn318vL97\nsO1kVpCYfaXL2QOBwPZpVGdNz7SEqVMuzgxJhoRl1dDTgEJaArXN4c0XS1lAolEUC24+/+DF+0lP\nv/55/2F8/sG3ef6a64/eP384rU+fvr1aHbsOyNpFxO7lk+Ofv+q42v7u9eUG0qxudXJ7szvRLp5v\n1tN9NhWyIzcfPtXNQz9er4dTVlZ6Ys7N1KI9LMslJxULLlk9mBNIE1MtFSpD1hejpCKrxmyd1SJW\nWMzO2fSSu6svn+V77O/X+cS0e/FxeX7yfI4P+IHlzWZ62v/MOs80WWrPz//p6TZ9OPv9b89ORU/o\n6OzL6UonZXi475CeqXCJ9QDlyleXvrFZiehlqS0iXXUd1V9qZskm6TIBFZZItmKtZOXQLQOetS2x\n8OQkAGQzpS+xTHP86PLC7t7X/jTmlX3zzR+dnJyj7vuHz/27kf3bi3UERJtrjJ/t8snOV09e/+Jq\nK9BcdB2gAb7ddKv1Bk6C4Vaz26zWq7X5dkXSBYsEKjWSbl2d6BGObAECixuHy2BtZI6+WTkBBVAU\nJiEjcV5yyXH7YeonaP7+3N/H7fRhf6/ufL3pf3788P1c9utvPvrki/G+XL7poVRa2vHsyT+cPind\ns4d/uDprD58OIczoQNDH7CBZPErROim7aiQsWYyJjgarmitXDPpi4Go+xqYqMZLIOYZ1+SE90nKZ\nP9wXod6yQ3zxj0fyzTeuj//j5XW13rZPyv3FZvKu//DF797YofyJbme5Mekx/+itro5+xv99dWZc\nIKPMrgeImKUaVjLZ0oMYhVWGRCYzxYyoGjiGy5W2joqytN6NI2tpYEggI4etZSoTKcHcc3HF6tH2\n3yR6BDAfJ2zOrn76Zz+62L3CulvPuL/99OW3MhzfPP0i3v7kP7y6/+zMRgNivjs7/eapcnP+f0yX\nHkIIUBhqmmetXljRMo4UKXrKFZLZYqchoVQ/yVSU3UrpFFKkLwaojMYjK2u3th/gT6IQC/fBBjm2\nLA20YExfnbjm/vDfXj0Z5vWU9rB6eOFfzYNLuS0Xm9/93fHfb97vWz85f/xWJ7F68YuvP++XX58m\nGDh3zYPlYQRpnsE0zMVb2J1X0MxURsbZPKzBuZ68OETX5tBWxVpERAtdC994oyZb41Fmij4/ooiL\nU3hR3fvKb8d18BAfnrJLf/bWz3133K1JFeXpby73d09y3GUOMu1Pf/LPT8/jye3fn/cbsgWwgaDM\nUyQb9kRTyiA6MrzZfRDhNp1VS+7OpjqUjFghWjou1Q6nZbWLVGjtTd/S2pP2g7GoDJv3bEEPk8bx\nvnab+Xo8O5tXR/uIDxf926/KSa2kdZo35cRN9SmmFKfjx+/r81wN/2114i1zryYiQROLZwsLbZhF\nQQKNIm3ZuOG09FTFYSPYOtOHCHoqJCEXXae1nMuYhh6LsAIy83bSRkiP0c7thDOKioriPFxvf3R+\ns/HNpw9f+z10cWB0JNXV2Hfe493p84djzLur9fcngef/snvWHmlHMKhaq1BbiqXi8aUxyw3MpVOq\niyYy+sJhDiL7oWbfZwtzbVowpJTJzCgDH/UJzUdrIszBx+LX5BZNPNKt5zqO+vyn429X9f6jN6/P\nhu9mP3koPZFS9Rf3Iw9x/OrzP5tEffR2ehpPDz8/20ApQ8DhTi00YgiAK9r8qVFEldKNhlQYHMEp\nbD5TZsYGc/Uf+CUu8YDW8uNWBcqW5UM+DhhLXKm3TwKzJmqrw2pXTz76KP7Xvzv/yeH529+dn8aT\nbdn6sRhCiuPDeQxd//Fv/unPzuPQnb3dnqxP/ras+4Qi0LKX6RAZTerI2sSACDjQCURNwVVlDphp\n0mGVBREbZbGx5fBqmUybZgKzD4shXi09OdNIt6W1VRuzGm0QxDx0L8+v3383//jjb1bX709WpTy5\nzt3z9JTqXC6fd/frzdmnF1+/w7j7dH88x8U/vL6Et1QiLPlM5Nx2J4nlkEwaJq6iRRtaBigMcuM0\ncHVUnbs+oyCXCciaaB6SkJUrbw462NJt0ZgRooja0jJaGgAE8r5+tP/qYJvTl+/fzMPqairV7p8+\nbI4HI1j8mPf9trs5/zS+2Ud3+Wp4evH2Z0+6cGsRGDQsaVqzGiaZWPJooBrWNQLbgKjGQM3UOIQB\nk1Y+t4C92qqymuMHZsjSafEEsGlyGyXYwjGXD05vZd1Y9M3r8ezJgHx/e1F2p3M38Xg8vdurTrUr\nsOH38aPh/T/819eb24rLcf54e//f1idER2QqwPZQoy6gXw3DnObtPVSYt1w+41Emt7COR9ZuHdVW\nipZO4WpcRetFqKxaORJAmje5MwirdEdAytCjt9cocJrK5QWO493+aj9f7ld59+QbDUdf1ycTVsM8\n5v7k9Gyz/XEZt+P42flXB756U04FEZnM5icxmTczqlHgPBvRuZncUDg1soVTKZmOMO2KlXXMtnnE\nkAFkCN4kkEJmKYte6jFqSKB1UAbtMbZncf5DlScfnu5rZ2NX+XQkdivuu7PDtH51kaf1YSU7/9H0\nXx4uvv/Wu/cffzacDIenKp0THMLMHEwZsmajpVMEA3232p7YJCtBdAzJyq4yjGnZlUOxeYgZq0hT\n64hkC8i4pEz31obwpjVssIIFSQcN5TF7vA3uw8X6+tuR3aCshROv5x+/Gx7Sbz+4f2m3Y3eq+eF/\nfFfrvuuG+fDBbT358vPXXxUiW6g4HlEkI7L+YBDuHLf30RdNiodKwOn1rnODVNPtqG7s53SL6IDl\nl7So9iU8xHsg20oLyRbGHi0SvEXIZkMUAJF3b3Y2oB4Pq6Geb+zubPX9M01n+6fzvp/P17fzdNy8\n/I9ffB0etyenGvrtz/+WgxlTnVs0F30sDIasuXd66O7nv/7+3ayx8r4zIK27j05UAWA+inNf+5Pm\nC8MSGS7gMak8jQ1Ra2+0acC0iJRMpTSvUC6pEBX9SjBXXzDf1vvps2/zTNHvjpcPq+8fxo+fvdN4\nZn/n63mszz7Y82r6nZ8Q1WgMWlo2X0n4I9oNoYyH9b/t1R+sXkwYMqle+zJMENKB7vVc2M/bdXui\nbDATW/pF6zO6pqUw+0M2qFBgIYTM2ToDEUQQBsjSS+a7Z/3dbvdv4/qJSg27/hQnEw4ffLza2eXN\n91ydldPh2/tP+/8FJyfdTSSt/CB2YtpisCcFpq0x+2hYPfnkcm7BZ+V6dhFoVj6NJ0HP4Zhm8UhY\nNMxGLa7XGUCbZR416klDWHGn3H4I5WtzB4nIetjiNjgffspfhB8/sHL69Zrn/aTXZR0X4zfPryo/\nXt3tT//8q3pVdnsVJPKxQDQ7N9jSzWWO0tlcu013sc5gCii6XyEhlDAMbjvPg/eWsAyhJVII0JLG\nJLOlgmABQcTGtBYlc6a3Ft3MDYtZy1CH8/XKJvviyTfeS5n9atLBn+XJ9dfHp/1v4/mwueLP54u/\n+u4XL7ady2U+eyN2SfNCkXACZlRCSfftSd0fKlIChl32DvWyWaJ3+9JN/SaqrPUhaN4fYglTLG2g\nhekHiJOkVc1JVCWt5baFWq6sKTWvTl8fNlP9pP/lwVX9cLvH5n0cT+KZHbw+xBdDvzr53ThfxP93\nbYcMdMYM9x84IQlNa6ca7bYIoW3BhqMNfN81l5Ukj2o79Dl00UpemxmaCKb52Vv/2pLd8bhdWrMO\npVtjZxp7TLQDYuou36Eqn/Mf69XA6NOAEVdz93COt9Odf4T7OJvfdeef/FedDMgqTIoZDERbSC00\nWrFgetnuWCDCQwLV308FlEVp2jNXrZOXimbtbh4bW0bURogt8p1F59Vcl48Ok5IxZ6MBla2Fs9TT\n27qZV6er97350I/ZzxHDnerhx2fv5vnDbv/tYdP/YuKf/vz15SboRV2BJUqlEUtMJrO52jJbHLH1\nxWh93yVQ/LonBZ+t2lxlBepgVuktWKCFTTUtCxDBx+tbGk7bbrwgrCYLpIR8WYJujbet8/O4K7N3\n7+ebdZlnGwqT91fzb8/8+S++m58/e953l1dfH/qfzL883V7NgajssqU3L4CLte/G2agHoRsGHvDR\n6T63HlrtjsV6IN2QXdelxiiGR8W8gIQvUnQ1dV67kaV9lsa2SjBKyUWb12DpJkkA5u36VRar77rT\ntSPqPK91Moy7L6ef7aaL84G/7X/82ZNvX/Hq+X8dhiyVKCVafmImCFO2iC640b1blTJ0zPHu7fe3\n7+/217c5lLd94QSXVxSvgGWg9o9RTI1RWlgMgEo86vO5gM8tOqkAYZbpCZloC9YIxK7/7PW8elmn\n9eaXfVU/HvDF3vr54eyTEXf+2Sfb233ZfP/NOv7knw4vOD43VMe8episEkJaC00XTb5/3a+GLgMj\nEyi+f384R32+eThs3CfKxYmYTnrgaKU0CDMXtz0Wi0T7xvGYaNpE7q0AFtEtYYZmVGlZfpIm+3x/\n41fdrc7j9uRkTi+HbnhYn13eXX75T6/G7pP7zbns99P8x+tfXXWV80qGjBMi6sCUS0waM5yJC6uz\nu5XUJHOzbW49z+z72VxpHHYDhNk64+6y92XkaZuAuVQKifJW0oFcojnaOiqGlAV9aVHUcsw444/1\njU6fvKvD0Va+wxrr4/Z6f4HuUr/eDri4mN7+8Yt/PpRPfvz/1BqwE5eXmOZuT0QHKZGGNNCUqxWf\nKc/d0z+l7IvucFNc5e3+zC3NSsb50a2I4tEX24tkAnwJyGocLNA0w8tIhnZLVC4KJDbFMJYYdSDr\nR/ot/PxhLF39oMyj9ZPPfbw/rnwcv92fPWHNaXzzm+FHP/mb+RxKjGlzyMMQCc1NbiZIyITFOM7Q\nNFYfttvS2WGqKPPrAS2Lfoch/Qi5LL1t3oWeD1vugniUri7XneBRNAKRKG1UFC1+6F4g3wyHb8NP\nV29WD/X55tjH+eyrs+qnuxN9cHsR9tTq4elf/i/69MO/+eo0jZZp3ZgeWVjJMIotlTSbltEjgyXr\nYQcb0Y1dZ93b+QyM1t16luidHqHyqJd95FyY4qNmfjFsNMCcP1xylFy+tniUBYEWh/HGu+HDd3k6\n3529H7uyud/OV7ubz3h69/r88uXZ+JRTeTh+8tnb3z1ZuVkpnaJPECzoxaxAiGbsHBlBsBhqIune\ncex6+nG36sxR3ATWGrTZFummWcOkiHbfVAuOXtLMHkVMjf5OhXKJsJJnaLFqIMk5SpnP6vXJPPYX\no1fneFcv9Tqerdbf/25bcXd7HNbjBx/v/mbVgVKkJUqvEuAU6JUzrQkV2yUSYNKo+bB7uLmbN9ss\n81src05BzxnuRiRFBrRQlo3A84YwAxIz2l09eDyNmuvEzeCIuXYFtrweaYJ3TK32tHd1Uw9D3ey6\ncXc1X73aTuXpeiwa57txeH+K/1K27YKjTHFCRtQuFUUF8YfetPXNUJZVX8r64rwfc3iXbmbF6SaG\nN7bbPPxRyEUCtOb4RrPlMpvrYZFXCXQz0s1SYvTFW+uYSGXj0m2mjb7pXh/72o1l48PuqqjYi3zF\n1fuyGYaPfvTz46mlEklzUOGUlYwqhJRNDC4K0WLAzcv64nRdMNfN/P7EEkpENJw55JbSlM7adF8J\nKEPR0gbaFGjN7WOPlzG1XW4SqVJq/FBWwEzJzO5xynp6sWNgVefucPdwOL9z3Bx0P7zENfLy77+5\n6M0pIzLhnMNIUZY2YyKbiB/NRGilMMyEuUaU9du02QmUIpMKkuIQxdGyGdtJlWipl48IavNuLiHp\nohsgpbLxI4CZmIbmDSSRKcsDwEqZpaviZEL1blr1F/HAYXOXu/tfXLpqCF0GTWFQqsKEMA3jLpII\nNZuFAXSHEGHuGHh96mC6Z05ryUHjrOOgRORCT7TjKZs8sMGAnaNdrSDS2ABRggar2VDhFlC8UBxG\nuMOOdtsrB6/sYwjefn2DC5QrWHYnmF99cmLmfAxwDzekWWvzxHXdt9/aFCs02OCqNb3LGO4PmwpD\npkW7MwjM3qpppi1i50WlsyBqYELZYbncqGVJKgWzMhiLkTZHuoF/iD5jQrGa9hlDqfRjqs7rs3WJ\nu4fZTvz96UNfDv9cJpGROTFbUo6Ujzth6kodR8m7YmYiFVOSbm7Kop2XsTNDlLDR4RU5U4KTdF9u\nsrPFGNR6LGSla5ns2aKazFhW25Vhlpgzu74dDUkCTjIO83StODskbQ3S72q3zVNMeew+qOOwfsOL\nPUum0Upz7ITREeZeGVBZuY5jTMejWkknRtIQETShB01mcA4yWHo63U3IKkZkpjfDa3OPCTAm++ah\nfiSVvAzbVb7+RckG3IOT2aOOTELGzfrL7z95O354OK7M5kQX4UPa7enF9/NJ3erijz46HO83BWU2\ni3noIfhsCdLUqcylS486V7F46VFIn1VyrsbZsTvMLfouMiRfwUCZVXswMKoXCloEhAQcSWV0JSE1\nyRFIuvn47t2xK1RC0bVon0VjwuD0nv/T5u3/6dVutx63tdsbgof+6rnz8Nn468/mTvfH9cuPvz1s\nPGJdEhFiLVRJwLKazfOqxLBKdAadmmQsRXRb40ifHva32wEQ2KXNDJNBEWJHxmSVJlkUACWSiWQi\nY/OIyKOFqefx+jZ4MhQDg+wU0RaiIUnOD/zP438/pNbzRV9WFf2uH/yhrN/1u6s4/eTu1R9d/OL6\nJ3/+p/94e+qYSkSBOsoEMUOJ1AjrB6MvJpFMX9wJWJ/2w1/ef3+j7co0KYwMSUzrsiIw181YaTLV\nph6Wtxse9IJSLk55ZKq+vfMNsTEZ2pUtoC2hSKY4nv6nV389n746MEo3e/VTqqvb3eb5w93u9NOH\nLNf50We/+P7iL8qkzrosxWFsN5tQDsLlCUjz/rDfTfOczJrCPI7TdNg/xJN/9x9fHG92KZL0ToiC\nSpcLUaxvbV5EjamGZAYzsPshSJKSr/qprgeWTWc5J8xowbp4YJE1zr785V+v538zukX6XLOuV9m/\nvnvy7frTz07XZ1/V1bsxd0+vfzP9xTi5FKqgsaKoI8IszVv4YwRIo5sECyXI4sxp9/at/cl//HB+\ndw9PscBQ5ZINiBi8791aHc65Rsw1pDaz5mMXr8LDu7H0w9CFbQpVUepcj/Mig4/+dPP7b9f2ef+6\nL7zuSq0xrbTSV6vzr95enj/55uE0T470j37s/7D+4n4MtahZsSJr6y1EVZhlkMp0JBc2Ei2ymZhu\n3saX/+mzh1f3U+RcJTNlHLoIWGRHdzeDqmpVTONxnqZWvwEB3q3iq6/m1QBwrrba9r3RKrjkJzjK\nELeHfv3kT/7H7FnuDj/OGfs1Tof6q6uP4qtf33z74Qo51PXt9RF/c3E2WWMZMwk6kn3KmajKKkvR\noKXxQrss1pmZxHz3/fTT//zk/iGKmwFZ5ShVfWmkvVshC5k1M8zzsJ8ehwxab2/fdxdbp2WahQ3u\nhYWt0GZKuasFebE6vt+WQp/m3e2N9zjV5ubV/kcvfpJlv96Od6d28uazq9v6x/ucMwCzwsjStUsq\nSuPiM2DGzIUKbvUc7WKXBOrda/77fzPdzQdjNJnRKuemJxTk1nWlc6OqrfoVK9mEwuY5ffPu5Hzl\nTWtikV4GZ8kQW0CogYj984vDvxRXnfsvT14+++z5W7w/xpf9r747G7/a5utua6Wzm198mX+7eXqY\nS19atK5kZigY+yCWHZK1XSrYJrMGurYLPmka3998+lf2bq920eusdUXX8PeGGYJWOrPiFrXvGuxs\npex+/couu3lMM+NkoBcvqJZpvnhVjGP58fVtvUiJ91//tv+jD76e7bD+6dPvho+/f1XiDPf9+cG6\n5//49j+/+dVPpgo1WE2JKgWZ7iEA6UVhPfIH9t0QVWoNXgrM3dvzv7KbY86zElEZamkzj9oEkegM\nYuisNJEhLO93p0/WUDPPNENHcUMLGMcif4nzzW3/sCpd56erVz//5hf3k9ZX+Yv7p1dv/AwPl+Pb\nKaI8e/H3L/78+7LdBaMqZS7zCnZJ29a50lS0hwMRSyfUarO0RN8B0Pju9C/9tnOs6OqGMGER3bIJ\n0OksjArfKBrRpPnATT+GGkdmJlYFV4MRPwQ+peD39pfHfVhX1v7kcu2+tduvfsfTuLs6YqvjZe5W\nvP3qj89+9aV+/aNjKb11nVEu64sVAF2nKJ2PY7eS5O7ZZB1kiwPJdqWjCE3vT//dm5s5MHdR/ECB\nLYpjEUIjkhlAWT2qD+qhbrcMh/U9EbCqpA/0iB9i+c1gh9d3P1Pxcdzrvsvrgz1j15Wr7fe/+3j9\nfvZ7P43eLbrT7377459/Os5TZsZUawh1msaUm1UJ87zuK0FGsNUBKNplxIskECDGmw+/vDkCMR96\nRmfUknogKtssW4qi9O3SXeP84Cc+5aQOcyQ7q82/VPqWbLiAdXbSnb/vCWlYz9FDa9tfdHV65sfD\neVcL3w6r+z3Xw8v89R9t1quRKVqxAlQzWPZkL+ZBm9IyrRdYqeW7ptgy0R8vzDrc/vTZXlktt5mI\naJNc4z4pqF3gao8sGh/GYahZrOushgpsHlkAoVM2e2ajpo5YPxeEOB+l0qHz/Ui8X50d6t2zXG0O\n71Z71Dte5s1/PzusR2UCkZGdY7Ap+kDJWd6j0oI0NjM+iIZaLhc7NygG2s9/fNx3proas/nUqZbp\n16xlirmyDwmElXqPlUYCBdMkp9l0D1DmhjERWvIG+ofb5+9VZEPeGaYqGbUp89cvNnE8nh1OP+Zh\n3nZ2vLFnr1/4iZnLvCXaxYza+ZxCFiRTzmgXwphh0Y0nyTYMqaE39eb80/s9x1zlkqmnPxjLxKwT\nVxenClEy7nMoVc6O4yS4VTM4mVZi2XALmOLv62QM+N4cmbHKCR7DdPvsfH0/xqxN1N3ZVRy/faaL\ncozZp+KSGFXssjAOXTGRGaEGvKq50RsEjhaKlWwHKjDtfzofEBxiAfv5iKcJyoih7xZbtdm463tL\nFxiCO1NWNImFmlDs8cYbUb7a+jrQJQpsAK8m7FYcuun1txqynw6jkTHNz9a7B487t87MYcwC1eQG\nByuJjGJtxkPCEIlsKrf2XTUEgw03vj9/eZ3hHo9hCTCoCeeJ6Nc9F3mtaTf3nM1hjKQZFuNQKGpR\nZrLdJ+lIfXL/4FNfZmTatF+V+z4n60o3HBldaru2uce17z65vce/TFk75JFKy+IBTYd+zYCLIbO2\nelJLwMFjEHhg8WVIUE7HL2dlV46dtPAA7V4WSFHX64bBgO7TwZqDr0W4MFFMU0LwgnbFDbmgKfnG\nUPoxU14f6meVKzw4UbXyUTFudHuYyqr/9dMvPh9e/T9OZwhBN81j71m61RBQEWAF+SgMtEUAhUVB\nogUbdJHI+w+G23mF6ovNBEl4C9dPbPkDyZSHuQcM5pZKOmAwsoZDcja2VMsti3WHsj5WI+1gX/L9\nXWJa1aojV8J8O233QzeebbZfvX6GS69dHWtxuisrte4VCPZpEcvFvo9+QtNyKTTVTiS1C8aBIz7Y\nzX20Rc3l2sWkMWst3tQBAG06qGTNlObZzCEVKzAYLNKbyCa5gLi73s926URWfdb/05e2ItdjDlBX\nNazYfT765D9/uv7is5/9tzMOp7AaUkFmr7IQHnVYjsxFNrQUpgboGRIua9ocQrH/dFyt6w8oG2Up\ntmi/FSIzzSDLSSs3qZRYCBpVFbQAQJSCpQeotEqPba7TKdP6g9116dYETDmwpLSN3/HZ9mHu5+HJ\n5fhv/ud77leHlW2j+pwGZIhlIo3WLv8NtDvdhUQ2NXxTRP2BjFTun/Y7T8LntKWRJCgD3YFM0NLs\neCymJMKbJ1wCUMCAibBEBLFM4xkXYW33TMN0X+zKqeGm67wX7HTuD7snH+1yvsq/+WDor748/Mfb\n4yTUCIkqnnJkZ1JYWz8N+pOnSrtYV0z44x3SLcTh6M9+3QUkzyZmASwYEFGw6LEsj7FCQEZlbfpC\nm0qB1SiAxaNEo+3DLm5hbkDgViceOqDrDltTZOTJN8Kk/d2zdxe+vt/+7PkhPt1/od1Nus3udIiu\nKVMlYWKw6WUfNeUtZcqWC4jQFp1Qx5f/r15SNWGJEmpIdqqoZcGG6kSBiEKFau/IIGjGqiU7rykY\nAAiZEZP1NSFgzfC5Ejv6MMaw5+THZ6dl3Bw7vPxyPnty3l/Ot//yt98/22pOa84u1zygU9P0o10M\nacwmp21XJyH0CH03+8vhabYitqw3EUzI6LXdCSKiZiFdSqqiLwLKcsVwk445mzRAKTB96CxDoQnn\nQ4Uq15Ad+mSZLx782dvz8Vt186v3H1y803V3Fpvt+fxVMJrKPY2HuQPjD3boRxWjPaq2Gt3soDIb\nL7PfXu5p5iIyvX2tSAP9MEp0AJwBtAljN/rAnCvkNCIqzdDk74tAN1mHWPuUqe1L3M4YpF0OZdTg\nY98dD1f3b/7sT07KVr+7/2R89mSWDN55At5FY7+OVpqCA6YMgVruy14yfwAiktkuARORmvKzo8UI\nL6UsGd4JIq3o7n5XjUkklFlVLFEKs90a0BlMEyT40I4PpUByLhosuX6yenu9eMe7Mvhu6/sX+XDY\nffD6H19Fr7nebz7/08nqXGI2ryZm506YdoN1i4+1LOVbLZ1MyuZEMWtv6wfCdf9CVuc6Z8rpZi0a\nyM173e/dSUa6Q2leRw2WWRuTY6LcotZctGiLJrAG5yTr+3fhHctMWL8fhzIdzY+y4eT5w/Wv3v20\n5/Vvtq9+v0r1LQFRFqYKdsfjWpaiUplGmhtsSQF65B8Sj3W8jSMxXnTHsda5VlWgOGFImtE7985I\nTNXcxA6Tes8woLg4W7u0rxSFtWyYNozbndMFWl8Q1rFO/QrHOSo2d8f1fFxvhq2/ezf4Z/rum9Va\nnAI16J2cIr08mLWWt6V/q/kb0HhFayn0aBEgejTqYu43r+eY65zzPOcs0qyQiDnZlwQxm037ufg0\nqys1MthBmg3ggw1mpat1mcVA2nB09CPWDIGbowuxL45K093m5PPD9HxXn5Vf3Ri7y3kdY99qgTp2\nhNQdr7fuVZC50aKF9GkZmCDZAmA09KDdNyBLrK/N53maj1GnecpI875n3Y91YxAneaa6Lo4YujkJ\nM6ZHGJjhmZmr5Q55E4Ts6yCH58zILmcPFqljcJV4+l0p9/kn9aY7++BF3nyD1YwuW2YOrJAwf6dN\nxyQU7RK3xrlzSedoOxCtrjeVThs7pgv5ZhgMmOfjPI11mgFVP18ZYaxj1LmWPicNqwxjWKlkylRr\nu/O7e5wrAUOZbjcl1B0sc9vt2UsK7zpJnX3Hbypv8OPuZHj6/p+73TBkmTMFyOCI7NfXt9viLdRc\nhsfoYy6BimiLrOn+W8Fimwun9XBEKZ27KWMad/v98fguToa5Lylk1DpGiX3tBhwXATckM3UdkkSD\n9JsYTixT2m4ckMqhOzjnwBA5x9rmWPEi773P2xf9+6+vttbZaJ0cGUm617Du+M5Pi6URXNhSLU0u\nmqbZzJbJQ5nx+JGIimeHIN1LVwqRwvHuWpcro0Og5ho2eGY3eJpnGkVMVbZZ9XY0mqcp1W6iIjD3\n63n0im4bt3BMfakIAqmprmKrV9fDdn6IixOfSlRUSEBfQMpX/qaerXxuSYyP8o4luq91c02prbay\nWhj3Usif11yGkDK4GzjVj9c4jh29+HEKW/UYE52mTASKKarJBvRlDrB6r8d4KkA55FDJdWucS9p6\nLOY8yvvDNKxWvP5GT170d3jWKQtSacIQ7sDGvjsOmy7ABkA3toHNWcak5eKVbLI7gWgygAQ0nZdD\ngGS729Dck4PyAT29HN5XutWjVoPXKkDWQRo6synpUCZLacmuTcSe5TiQUe9nmqCY59p5JUO2mlC5\nutx996pc+v3brgr0EFAijVyXV+/WV9ta0eBWLd5Otps9HpEmW+Jrl5LRsn+ksZzumqMhEeZFKtjP\ndXXqbvffzuadT1EGG49CReeoYqlporWcIlq0cRNI82OJkb7cUd/BeWCdDsOpeNyursn5STfvfjH+\n9P6mwF2Ay0td2WqL37w7PRmiPorsWsPWrhduZ6EvauDWV7mWEFiSVMzPpzloLdI6WKzPXWq7Vdy/\nmXp37DUMrMvtnj4lNc99sZLIdp/gIlwnEWSxmfQE+gMigP4UQom6qny66jdxPOm+eP77J5dbEVDA\nctC2J769vn9y0SuWjFfAcpnl2ioCEKKDAqMkOTdYo8XzE8cnqCUMaHeq2aqfn1ln9bjfVZbea3Qo\n0kzC+i5CBVOxYl09WAqo9MTi2yfteCHM7jYRKuu9Mc8eKO8qp6rDR6sc+M2Pzq6/GGqxOYrm9YY4\n3N7F6cWVcgnoal12UoI3q1lSgrUQvceLr7TMRCQD43Z1vFKzb1hLq7NieXg7mnHlMfcrR80AC5jB\nkmMOXirq3Kcbm8O8zY6pNSa5myazpIzFJsvIcgrev/xWx0//xTc2fdJfb2TJHr3h/rjT/mJbfF7U\ngETLNJHbEkaVMgY9TZZL+DGt2e6XUBRUPX0VtqCkdJphCh1eTxurayq87zjNVkYOYMI05YrVpplO\nWST6pTKRTm33d/Q6Jy0MlkLOUeHDdo3Qk/F9PMF4er39dJ6paszY39/p7PkH585GwCWgyMcvX4/x\nAGqTH1vJdUDt6puFBzBpfjLVgFqoL43eH27ffjeWzM4wqfOcxlpnDJ3TgFCpUS0mKrJmnWkINd+Q\nuFZ6Dn3NQi9aZ+JEyNjT71wfTHF90lfo77oXc+n7mGp2JxenW5egOtfmtEwT2CrEsu+0HLoA1GRE\nC4ZjtviYoPG0m7Dcn0yjbMDDwVarbmWosOJRVZTeZyB8AblsYKhpG4FcTlu5H71fqwaoanzoPI/9\nfSSOh6e6PWyf7h50etidfnATrDFH8eJuVszVelipOY1MpBsWVUHb8o/nyKP7VrVG1ZLqIkzdyb5p\nQtK9iVLNi5WOqlmKj/NsbnSgQ2qeg6hpQJ1nWw3DeV8X1sSsvP3dNAc8zRC15HQ6z6dG4+q7i+d2\nmC/WD7un22+vh/w9qK5HoJiZECliEc7C0LZqJB4D7JsgE7ZQ9ElD+6Bou1LImufHhDeFsBINZmsQ\noXlX5zGipnd0y0U4rDQFgb7LwAkXHVYmVi9PplxlyqoIv95WnVWgrKbpw3z3ffb+8O5HvjucXwzj\nNKe6wQ1ZG/AaGSKMNRM0JAwLVbHEDT76LyDkopkSZWhWpOlpzlBtgQOP9zH1JsCHggQ7S/TFNAsZ\ny9VrcCZXWTX1Hu3PAPSTIlRYZ6Lbus5rTpNbN5Xfn55OuzNdYL45p/zig3yYKvt2G187Jts50Qhr\npOhtAgekWBJf8WiJcao+BrG1UVCHk+4oFYFtblQQbm1JxbGWwcTSOSAgVLOary1BTGp1VI96Yt6/\nGXtNBUiyeFfe+dVhLEWy+c1nXs+GB/ah4+3hnf3pVuteQXohSFeQhY8MJZpfC4sWtfljGrieATDa\nBmkAYdMX1nI2Z86VyDAH4N4NyFTJaQwQUumNIuo0yT2zK/9/G7jlLbgYMMgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=198x198 at 0x7F14B098D160>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 输入是一个batch，batch_size＝1\n",
    "input = to_tensor(lena).unsqueeze(0) \n",
    "\n",
    "# 锐化卷积核\n",
    "kernel = t.ones(3, 3)/-9.\n",
    "kernel[1][1] = 1\n",
    "conv = nn.Conv2d(1, 1, (3, 3), 1, bias=False)\n",
    "conv.weight.data = kernel.view(1, 1, 3, 3)\n",
    "\n",
    "out = conv(input)\n",
    "to_pil(out.data.squeeze(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了上述的使用，图像的卷积操作还有各种变体，具体可以参照此处动图[^2]介绍。\n",
    "[^2]: https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "池化层可以看作是一种特殊的卷积层，用来下采样。但池化层没有可学习参数，其weight是固定的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pool = nn.AvgPool2d(2,2)\n",
    "list(pool.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAAAAABVicqIAAAZt0lEQVR4nEV6Sa9lWXbW96299znn\nNq9/8eK96DIiMiOzsquiqih32FbJNjAwMp5YDJAQA5gwYMIY8Q8QI4QsMfCkQJjGSBjbYKrswmVX\n48rKqsiK7CIjMzKjyXgRr73NafZeazG4YXMHV1dXV3efs9c63/qazd+hmrnUlfRPPmyb6KSHOkYJ\nhCGAhIkL3IsQEPeQaTQ6SumGCzvj0FFE4UIyvT6xb789rkg4hRpd8952LDQ3GAgdvBQxguJ0E4Bu\nQndXKpywoIRR3d0AOBzeZQ3B4AQc8O68CdfuqIoQhNEZ5eP/LTCQjMFVzeEW6Eo3pxlIVwNAd4cT\nBvD58iwghXG+zB5iTFFIV9UHHa69nIu7uxkIxsd/+UTcCUgFLYV0Ak44ACfhpACkgySFBBhIgrBE\nCCXYIpeYggQJIBznh1p9eVrUGSgMVfrwRz3EhUIR10IhXGy1isEdbu5mZg5XLcXVzZ5/62YwBuNQ\n3Al3GAE6Hsx994tubu4IKXx4xxBFQISKXiBW1I0AHCDgZgDc3DzQclZTs1KccCfdQRdhKPZ8a0gR\non2cw+u7xYXu8Hd+ko0iJKUKdAfIQDzfLJiv3kAXV4O7GszcHauqA4CCsFLMnQ6nUISfner6L4Sc\nPQR5/x0PiRARiWJFCTMtjmIA4OrmMHNzBWHmAlgpZv5X9wkDGGmmpZibwwnAkD8ruHazL8rw7k8s\n1cQgAGPuB2B1eU7CHSThBqGbwBxOCVKPa+SianCKgAZzumkp6mYGdS1FcXhk1VeanP3Ddy1Fc/VI\nD9qWMHGKCAxiXD2CIEgQEHNvQorjtRh9dvy0j4xiBqdT3RwFoBsBteKGcn9jevFvvJU+umMhktkl\nepB2kddIAOZugAACQFzECHf1OGlymFZMqpsXvnD0wcID4W5mcJA0JWEGV1PQnz66WX/l8Tt3LET0\nJm7CShfHbRSSsOKqVuhm7mqmbjpaH6nYktL3Q6uBiFd++WbOpZhbMTUA4l7MSynmDnfaRwvbvHG7\nMImbEEEqdiePlSRI6qBqMFMzdbhbiuXs3HVou0E9u9TBLcc3fyEscimrX0KEbsVZ1AARhtDd0/yd\nXhKZDYUiKc8fn8FBqJqRgTA3e35VcrToYVB61/aMLE4Cvv1r1+bzvpg5nRKCl76jAAQgkOrp4Te+\nFYUoknJqTqO2x08DSHeHggI6VkAlxrXT0EWJFqPTy9COEqMIneFn198/HkNpihDE2zZILMEBoVDC\nf/y2S+2DN7kaH31JutkjE7oLV93kTkJWkMXSZqHCtMDc1VOguufM8cbX/+nmbMjFCCejHz+ZQ8xJ\nAowP/0IZvViTsT5c+jVZPjkVAQCYklSBrOCREHlmVQwciguAwd1d2Lb11lrQNvxGbrWompsztPee\ndgICAKT907lX0bUaOEm7X09y9FAC4e5cbakJnAYQkHkWUtUDA2NMpiUvziYHI1VV7a7++rwr5k7C\nQ3z62bnTAYD8/nlsAkrtNpqs/czmXB71CWbucDoYAszoJN0l5ySiRauYWNWelXaEa+MM6xXU/LWv\nLOeDEoCQ5elxK3CA8cHtIYp3Il7JxmtbeV2OAoTmK4A0dcIFDoCxje6Oqo6Mbjm7leP9XZXS5oCS\nfci/9Q+sVQdIQTi7f5IJY+z+LLMWQ9WH0Nzc9kktCGAQ+F/Du0AdTht3fQkwYxUnElBKWcqNDe3m\nAxJDCKq5fPXvzAZzdzcJ5eGDucMpbz2NyV3rPkJe3S+xgUTA1QG4rHrLhUJovroQFEqMSNq753m5\ncZ2tZaMIDKpa3L/+8lIdBBCku/upX6iqB7dDim7Bg7QvvqjV2CzC3N2wGigr8Aows9FahkQTiZUK\nbV62Xlpf1oOOaijEjBAUn/76vzWYQAh4++Wf1+B/lFOCGDUtrn3JwzoU0Shubi50z6pOhxmsvLB0\nklECS65k1t46sCVmo9oLkZkiPUez5a2rnQvdNeeL/+xa2Ujpq+8DyLSo06+O8oYoXEIIMThEBHC6\niASBBLkyz0DxkPo26qx/Y7/vB29ywJB7iZ7V3IHBfrOYCLVdfulfbaXpMOteA0xtgPibl/KEBY4Y\nGGzFSUIIIQkoCOK+cVe8b2LoMJbWX2nmIg206dmHZjV3hiqItFe//qmKLvt/9IvdXmpzTfEgnYcw\n339Nm8rcTWIkSMAhIhSQpAJo4jms9t7q2GfexJlMKg3ehflIvE2s7FlT04uWW3JY8sVfqO2CtRYT\nngZjzMHwlVFuHA5oJJwxEMJAQuA0V+S9ZaxtYGPWYrRjg05cBmvkdGreSq77oZEhGGTe7nxev7je\nVql3CUJ8EoMVIN26lms6aO7RxUhIEAbCYeJG47CPJngIEi3KLpc6jn46TUuMVHqJaR7qXPmQgyvT\njY3B15FdJAO85y4ews4blAR3AJRVC7sI3OkON7i7bVq1NSIy6Ttl6JswW45tHpKxYOj7yL4u/aIt\ny/PMYJNmKArmZ6onH6k5U76+YzWdpDBEA9zwHCJDoBgd7rGMB+37SZF9xfl2MJS5r4k6tIvdEOrY\n21APZ9wWroUBdAl8tMy7d5ZJMmz/utVhxYDEBWYGEZiDWJE0QEuwECduJewN+WQqfWZYxq6YLdT6\nbFh2XTssumYD44m2mUIJZ3P73L5lUSHTL048Pcd2IAJwrgaKCAADYdARYzVMtd4puthOfW7CMM1h\n2YdUMDIbMHeWZttHsYNIiEDEY7RVfj+pRN+7bBVdIeZ0RAQLFR0EaWpOC07VRkbZh/PNnu2W52UK\n3drJPI/RPHh51Fr081TZfkgj7z2JRELi4/nH7a+8PZ+6hHAzWsKqwHCPhAiFpK9IIyMImFW1+2i6\nqBfbyF0Vl9PTpcw+ib8aP+05OZ63p7+1m9eawoAQIE4++cH78rP7/7pyrcrBQRmJwelwh0cGZRCh\nr+iiwwkynI/XHh55jic7krtKuumJ8lTH7U/LXYyPj5b1L1bVtmW6B7rTP/7Jw3byxmv/+agi2L8Y\nWIEQd9AdMdCxamenkwBocD68ao7JcL7R5LZiHs365mg4PGIt69XD2KW/d2F7VFYFZAjPvvlA1196\n+cq/eye51rhwYGti9NUyQIwWXOFGBwS++hCqD3/OpmepH0+GRc1uzeejD878tH01/vhKvxxP/v7a\njaqrYSJCefb9ki7v7k42f+entRQP/kqDaiWnnE5oJFalAAA+ZxMu6XAZd8oT27TjqffVUSg/HvJZ\nmeCnlzbvLa//avMF9KF3VuDwR+9eDa9MQ9r43e/VwS3m8bUyJVasAnSnQOCqIEnC3GHmCCn95Thv\nnm2U0zXkCt2dDy2fH7TbH9ni2YWtV5o3ksEHNfHhm2+txXRqo/jb366iwMkrwRvQSQGFhAtAz8VW\n9JPqBCkMox893YqjPNSBkhfHqbYn07K2QN2Wza/pFzBkU42hWP3iVjqrS/Po39xJDX1gGF9aVgG2\nQi2u/s8dRd2dpJg7aABjCnfXGilLWXgoTx4/Y197XR/1i2U6Tde75bzVIrlpmnjl4OH5o/Tw956w\nkugabHvDR6u5T4qDQDTQzZ9rLAmBhIOCVJrj4zSe7/jJe6VdxGo9VzvLvLy+d+HF7TQsK5lN1mu6\nVLu4f0VtISKIA51XpYGt1K07CK56YIUqTllpGglgqI54Nw9lenR4cshzOVPWn+6Nlld/7o0X1Nla\nnFejSKma6e54ftqzDQHimb61z/V+MSjcV8V3xOdiF6S4CBnopru5VCcnfZObMHt8JKVJ/XG8VC8P\nL/7yzv6isHg8aeokZJDqwkY8fnRx1IYQe1T5YkrNoB2CiAhgDkSaqa/AHohR3MnQeUS5vTHb5Ozz\nuS+3BZK6Nuver0z2xv6sPu+HuhIBhQjrm+OzY6laETdBvOmTyGzmvTMKHHBRd/pKgtLMAQqliJZ4\n99laHD4405RO9vqSJs/KO39rcqFjM1qcaaoDFKrqmOxs+LknSvAcyuUpa0McjaoE7xftYCvShRXZ\nptMNgMGJOLl4Wnl3No8NuT0bwrDWnH0p3BgVVuGHRzmmaEPRUooitT4fNkYSMoVXvKK7e4ixChTP\nXTcIKEJ3CCBOPrc9KHuPv/Pu1c/e3z6/1PGsjzb1buP6a+MU0tGH8vhP/uTjbH3RIfcP/u+dTJ1t\nmlgBmt0ycTXN2Uxi3VT0vIgQFDOHUSCkUJ3McmBbX2refhz7gwcb7eLq/TIft796c7r2+e6d++P5\nfp795fx6rdFP3/l4GO31pVsrSU2420h0d3cXuAvpVjxCXYsDwkJRNQNM93clxpe+e7Z76eHupxfq\n46MXP5odf/XC1pqsf+etrhvGy1dv2dl0Mw0//uGztokqQ53YO7p9GeGvjB+H0dSDSclD3/cuAMyM\nrqp99Utus/vfPLyK97Yjzprp6fyirL10TRr84M8fd+36P3zj3sz7IpPv/Pfjr+0vFy6tjFjo9UWZ\nrNwrN4eV0ucua5Tn0oESwBiTQIZP3n6h+7OPFtcfjXzcIT/bap9mvLGVN+Wdp5vdK4drr974b7mt\nMT7/5JLvlwcF9VBPjyHlwlpY2XzidDOzknNYjwLj6mGkhCqm6B/Wr+1t/8F7wxufhcmmbI/bpSYf\nbly7tDF6+7MLVz6uD26V+Y1ZKRg/2j4Y7d0elVK1o1BUeJE1zCzEImZqxYpc3JxGYS6ucEpY0fl6\n82+W9/7Hh5cO8mwzdKN0cFeyZHn9cpr88N5eHV/uZw8fHGwuo+ey+dLR/NvnFx+59BueEdIVrc0d\n5q4oWhTbu5OIqHnQbjAAJM1M9JfO3rLDapHSeFZdPmtKVMDeXN/bvPPWpTSOLFuxlqFin/Noq9Cn\nZ4dZS6UObcaB7kYoYGZFru5FN8S+5FzM3Vdz3sLoWG6+55f2lt2z6Xa7OU6jZZHJrUtrh7//QnBU\nVaXitqwzhzytm36MgRFt0sKyLw1tZd+alcIbF2N2SBxU1QCsTFYbb6TZu/l8I47C+6n7+ODz+38X\nd5nfbK7w309qumkI0X1gMil91CBMwYJ0ZZyTHkizwlt392JXL4iTIcZetbj+tVfZmS42h5g63svR\n20fBhhvvaX312vQ/Pd0W0TIMManmYq6pD/Qg6NRkOWx5hc2qHoDVvmjZ2A9GSZFSDJodZmqOsp4n\nm0PTcNmnUZ1KHpW1gK+Ul3nzs2+NVAHzoR9US26z6mI1r+dDRFc2abtNWk0mN9MSrtTurBpxEYFp\nWTE903AQn85jN9u/FLaK9WF/49YXPjrxm5ftt3Ws5qamXTsUy6VY7svI3MupV8j9BP1+qNQcQsLN\ndjeBMKqEjKLQXIwUF+IsxYHzfmqdLwvH0r4of3i//xr3/svjagoz10KziL5vp46+1AO0PWIwnU/H\nw34IK7YAQsNBUtbRXCgxIIRIOMiwvbnQszBZhlMs0uVY9/bO4s78hd945fjtIDUFBjM3Vc8hS2kX\nCyvl81agPK+m9Vak+8pNdlufOFMwBynFJEw3K8DBcH5oGaJnBz7YxY1uI9298nloXhvv/oejxLpx\nw8pYRxjz84fHz54eQ/P9EIKHZ5hcraK5O9yKuW9GxIqMgYju4Np6glLgEFRnk/eufpImJV18e/Pm\nN+flZ1548uQHT3YW3nSBZhqc4jJhNfKXmlPPj+eMI4tHGO8h+mrw0SxMIUkcdGdUh3otDoEQLhvJ\nbj57utPw2cEtf7LV/e3LH8jZW2Ote4+BJIskA9jUV2VjyFruiddgPOH0gothZbO7pwohAjQhpZRs\nNCcdhNr2mV98OpvY2cmhXpl+uvvqHu6v/25US6W3Qrq5dkOfi4zWpxGDHZ4giJt1w2QUVvGNO1BS\nkooQJ4HYe4IbVpGBXzgZD+0MMmUbPnj5k6u/sjf6X1e/W3aO2HgfBgvuMLQxINq9zT4fV+8jMCCF\nTseBgDudbu5BquCEiBhiF2hBJES66w4wzQ+YlmuT+fn2R9M3dl76r/X8ETon2MfOUChV+zFGacij\n9aPR2uFhyBPz3CyXG2klpUAjLKVqlWuBiO6G+rkkmsqnrx01dnxhujbH7lE8aGbv9y98o+4KWeoe\nbe+BUo2vt5amu83avVK++dwVrTmsizgQFE53r+tgJEl3jwYHi7sANe5c70/X8tHENssz3cV+Xr/7\nxW+UlDTCUyFaqyhDrG3w7lQeI75/FCEamBCtdkDcxRVummpxhJXvLaqZKQ8GsJ/p9snTyaxqZ7vl\nVldtPpylV/7wSEiKqPS2OGv7Yrkg1qGcnDEevueeaktCRpUVewugmxVEShDkghCjq21Mq+JGMMoi\nh3ZedViOm6tPriy4+cfvBY0O62NcSjmlJINJasIk5dLd7hliseRCKVUWN3cC7mpB4FATMXhUQXye\nEsJ4NgwTicPoyafcGOzgo+99ly7RgQxK9uX5gaYoMU5iQpr1jxgTiLoUrW1VCwfdzDGVohBR9ZUV\n1fVR6CRR/HG3E8UWG90DuTisjX80CBgYNGgQZ5h/0mFtSg6JinyoqYYMIZTARNeOTndTc5UpDUIt\nunz0bjTqIi/pRmh/dX+4fxJtUHnx0+Vivb5664+fhroyybTciOfdzk9N1WKLUnIfCDoE0cJQ94cY\nVSGKA6WMpmpQL7MP/uynOY5sWc7rAhjzC/hztjtDHreznc8/fFMe8s5v/v7DldUGL8G69ZsXmhiZ\nxBZhfOmNr/zFu52WBgayXvbaUUJT0a3sVYPSyuz27x1Od+OLz8RmYaFuYfvsT6/9qC95uPn4J1++\nsHfw9nhjsnhz5oWRFrwkJ8ZBjVEHQzUNzauvPf7Wd/vkRel13kDfd+UkBpEexYRY3P6D053tKK+9\n+cJOPmvdGbsnr2CG8wuzyd7RvZTfG9fVXvfuGzY44JG0EQfNRnMUDXWSnO3yP/4X13KXc9YQJuPJ\nxtak6s+O/eTYIDZ75/e6g61KRC6/ernJIFGGW5MPWJqtn+kX23gwPJg16xvv7Z9d7EMIZEJskjto\nEsUQk5eSi9oX/uXrx+1gfZIU63q0vr65NlmvdkISe+cbabder1zacOHqhoNRiKvDrF6z248+Xdgn\nG58EHHeyV23vDwSDSKimG22UXCVzpkjt+5yL2cY/v7woLoEEQ6iayfoat2+E4Cc/ibvV+hoauX02\nPthxUoLIk/vLbCpPLobDnXf3hiXbxSXIhmT3bOjL2lYuOg2gC91yybmYmq3/k9jO4io9EYYQm2pn\nR6D3n1ySeqOwkbuP02hzcILkbAJMFirrOc7Gi63dxdEQ5YdXNsoqEdOmztbQnUKaWlY3czO7+hv9\nslFx54pG+ORGQ+8+CKGZdotaZDjxSgZzgvHMY8zFZNo2C+SFTM9SOMVOTrES7zk2VpQiMMOKqKkX\nB81+bbqYiJCAmaluv3y9oj0+HFU1lpEmed7DzQgKtp5K3Wrejb1NZs/aeT337mDz8XZyDSqy1kyj\nhEjY6uXFDOpwXf/FYbpip2ZZ5fWDStB+kL2ue68B8aVKTSXgXM5GS+fowmOItnX7VDYD49cflLhk\ngK5dGpc6mAhBNxhUV6cC3PTnMQGeRyQ63kmRevgkMea+EluKF/WmUtgwFK63CKPRR1ZTg9R867o8\nu/tELl/ZmjplfSQ5peAUFjM1MyvqbmauN/Yah6k7xTkJg7P7RELCoFGHHNl1UgU3JobQimEQPQl9\n0+a1tjmDjn94ht0tn7cbGzG0kxEDVla1azDVnMzEvf5iQ0AKAMZQEO34qdGHeQyd1ZL6VsPIzOHe\nNMhpV72lIAi26iiPN65vyfGDjwefjmGn0yoF+f/muxVdlUd/Pg4ZTCII1fLusejpcsg400lXAoWl\nLXFi5gb6JKTR8ZAYG60mZyW++8A+uHXNtM9DvTbivNuIpJDigLuWbJrNQPjr9SefHrUmkupp+Pgk\n5aO81DxMy0CcRw1DaoKaC2QuQealif30LGtMa1e/9/7W+r33XW2YbK6LP2w2CIJ0WRkC9JxKcAc3\nl59xNJ42TTNtTrHN88PT2fR0HYM058eRuQ+jaTEHuajJISVVbXKVy4CffX/r5MfI1DDamsrZJ19e\n+SYAXElQNRR1DQLuNjIsHyNMb1za7zbKxx8/XluMxi3q0vayfREyHtFhDPMyWBWHQvdaHffvb7+G\nqh3ENK5vQG/vXgiEm4is4lq3XLwUNXV/Ja5vraXlw9vf276yWZ/d+RTFtvrBYp5bvInG0ubCKWIh\n5BS6yioVM4nxXvXK1INaqbY2Jrjtr1R4Hh2YuCghtCGYqSgOxmiE9JNh09e6H3zf6nJxOJdJyqkW\n3Zo6xoQLWZaiQwjW18MFGdfV+vFn3QMPLtPNtdn3+9enz8WaQ1Zqqrjr0Gd1eLrWeYjTzYOd5Zb9\nn//Z17YTTrWedpareLo960OgEc68LA0hEevtxixXEm3rrpbA0MyP/GB/ugrYAIqYKE3MSvBStAT3\nl36aYgrc5nD0p+/mTVyYHM0m69QZLJ5+jn40CgoRyUdBK0kopR9d/+kLH758PBw8jaD0zeXtSYo0\nBTU4QVAIh7karAQVv+a91FrX5Xb/sKm78ejkzHYm2g+5j+Gsks3dJyTQPwvihI6fpb7diqPp07WH\nEzWm7YOtKiXh6riB0whBMBe4uWaREIKtb8zAOoHzfjOcjMZHC9najOenfTEp7XJ77WrtbmaTPohI\nZfUkVecv3dteVJ/bFcXW/kZKFDcJukq8HCRDogQHTM2sQG7kkoMEidOdkCZ9CaPLo+XJaQl7Yrq0\nAiGI/txNQtXkF2e1HY5Hm/1MH9ubO9tVCBJAcSdWh8ugcEEg6aZqpup2qxmGITTV5ELs8snQVDsb\n/uzzBXYP/h8CwMEc4WwTQAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=100x100 at 0x7F14880549E8>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = pool(input)\n",
    "to_pil(out.data.squeeze(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了卷积层和池化层，深度学习中还将常用到以下几个层：\n",
    "- Linear：全连接层。\n",
    "- BatchNorm：批规范化层，分为1D、2D和3D。除了标准的BatchNorm之外，还有在风格迁移中常用到的InstanceNorm层。\n",
    "- Dropout：dropout层，用来防止过拟合，同样分为1D、2D和3D。\n",
    "下面通过例子来说明它们的使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.6993, -1.1460,  0.5710, -0.2496],\n",
       "        [-0.1921,  0.8154, -0.3038,  0.1873]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 输入 batch_size=2，维度3\n",
    "input = t.randn(2, 3)\n",
    "linear = nn.Linear(3, 4)\n",
    "h = linear(input)\n",
    "h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(1.00000e-07 *\n",
       "        [ 1.1921,  0.0000,  0.0000,  0.0000]),\n",
       " tensor([ 15.9992,  15.9998,  15.9992,  15.9966]))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 4 channel，初始化标准差为4，均值为0\n",
    "bn = nn.BatchNorm1d(4)\n",
    "bn.weight.data = t.ones(4) * 4\n",
    "bn.bias.data = t.zeros(4)\n",
    "\n",
    "bn_out = bn(h)\n",
    "# 注意输出的均值和方差\n",
    "# 方差是标准差的平方，计算无偏方差分母会减1\n",
    "# 使用unbiased=False 分母不减1\n",
    "bn_out.mean(0), bn_out.var(0, unbiased=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 7.9998, -8.0000,  0.0000, -7.9992],\n",
       "        [-0.0000,  8.0000, -7.9998,  7.9992]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 每个元素以0.5的概率舍弃\n",
    "dropout = nn.Dropout(0.5)\n",
    "o = dropout(bn_out)\n",
    "o # 有一半左右的数变为0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上很多例子中都对module的属性直接操作，其大多数是可学习参数，一般会随着学习的进行而不断改变。实际使用中除非需要使用特殊的初始化，应尽量不要直接修改这些参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1.2 激活函数\n",
    "PyTorch实现了常见的激活函数，其具体的接口信息可参见官方文档[^3]，这些激活函数可作为独立的layer使用。这里将介绍最常用的激活函数ReLU，其数学表达式为：\n",
    "$$ReLU(x)=max(0,x)$$\n",
    "[^3]: http://pytorch.org/docs/nn.html#non-linear-activations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 1.2836,  2.0970, -0.0456],\n",
      "        [ 1.5909, -1.3795,  0.5264]])\n",
      "tensor([[ 1.2836,  2.0970,  0.0000],\n",
      "        [ 1.5909,  0.0000,  0.5264]])\n"
     ]
    }
   ],
   "source": [
    "relu = nn.ReLU(inplace=True)\n",
    "input = t.randn(2, 3)\n",
    "print(input)\n",
    "output = relu(input)\n",
    "print(output) # 小于0的都被截断为0\n",
    "# 等价于input.clamp(min=0)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "ReLU函数有个inplace参数，如果设为True，它会把输出直接覆盖到输入中，这样可以节省内存/显存。之所以可以覆盖是因为在计算ReLU的反向传播时，只需根据输出就能够推算出反向传播的梯度。但是只有少数的autograd操作支持inplace操作（如tensor.sigmoid_()），除非你明确地知道自己在做什么，否则一般不要使用inplace操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在以上的例子中，基本上都是将每一层的输出直接作为下一层的输入，这种网络称为前馈传播网络（feedforward neural network）。对于此类网络如果每次都写复杂的forward函数会有些麻烦，在此就有两种简化方式，ModuleList和Sequential。其中Sequential是一个特殊的module，它包含几个子Module，前向传播时会将输入一层接一层的传递下去。ModuleList也是一个特殊的module，可以包含几个子module，可以像用list一样使用它，但不能直接把输入传给ModuleList。下面举例说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "net1: Sequential(\n",
      "  (conv): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1))\n",
      "  (batchnorm): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (activation_layer): ReLU()\n",
      ")\n",
      "net2: Sequential(\n",
      "  (0): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1))\n",
      "  (1): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (2): ReLU()\n",
      ")\n",
      "net3: Sequential(\n",
      "  (conv1): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1))\n",
      "  (bn1): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (relu1): ReLU()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Sequential的三种写法\n",
    "net1 = nn.Sequential()\n",
    "net1.add_module('conv', nn.Conv2d(3, 3, 3))\n",
    "net1.add_module('batchnorm', nn.BatchNorm2d(3))\n",
    "net1.add_module('activation_layer', nn.ReLU())\n",
    "\n",
    "net2 = nn.Sequential(\n",
    "        nn.Conv2d(3, 3, 3),\n",
    "        nn.BatchNorm2d(3),\n",
    "        nn.ReLU()\n",
    "        )\n",
    "\n",
    "from collections import OrderedDict\n",
    "net3= nn.Sequential(OrderedDict([\n",
    "          ('conv1', nn.Conv2d(3, 3, 3)),\n",
    "          ('bn1', nn.BatchNorm2d(3)),\n",
    "          ('relu1', nn.ReLU())\n",
    "        ]))\n",
    "print('net1:', net1)\n",
    "print('net2:', net2)\n",
    "print('net3:', net3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1)),\n",
       " Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1)),\n",
       " Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1)))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 可根据名字或序号取出子module\n",
    "net1.conv, net2[0], net3.conv1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "input = t.rand(1, 3, 4, 4)\n",
    "output = net1(input)\n",
    "output = net2(input)\n",
    "output = net3(input)\n",
    "output = net3.relu1(net1.batchnorm(net1.conv(input)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "modellist = nn.ModuleList([nn.Linear(3,4), nn.ReLU(), nn.Linear(4,2)])\n",
    "input = t.randn(1, 3)\n",
    "for model in modellist:\n",
    "    input = model(input)\n",
    "# 下面会报错,因为modellist没有实现forward方法\n",
    "# output = modelist(input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "看到这里，读者可能会问，为何不直接使用Python中自带的list，而非要多此一举呢？这是因为`ModuleList`是`Module`的子类，当在`Module`中使用它的时候，就能自动识别为子module。\n",
    "\n",
    "下面举例说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MyModule(\n",
       "  (module_list): ModuleList(\n",
       "    (0): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1))\n",
       "    (1): ReLU()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class MyModule(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(MyModule, self).__init__()\n",
    "        self.list = [nn.Linear(3, 4), nn.ReLU()]\n",
    "        self.module_list = nn.ModuleList([nn.Conv2d(3, 3, 3), nn.ReLU()])\n",
    "    def forward(self):\n",
    "        pass\n",
    "model = MyModule()\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "module_list.0.weight torch.Size([3, 3, 3, 3])\n",
      "module_list.0.bias torch.Size([3])\n"
     ]
    }
   ],
   "source": [
    "for name, param in model.named_parameters():\n",
    "    print(name, param.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见，list中的子module并不能被主module所识别，而ModuleList中的子module能够被主module所识别。这意味着如果用list保存子module，将无法调整其参数，因其未加入到主module的参数中。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除ModuleList之外还有ParameterList，其是一个可以包含多个parameter的类list对象。在实际应用中，使用方式与ModuleList类似。如果在构造函数`__init__`中用到list、tuple、dict等对象时，一定要思考是否应该用ModuleList或ParameterList代替。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1.3 循环神经网络层(RNN)\n",
    "近些年随着深度学习和自然语言处理的结合加深，RNN的使用也越来越多，关于RNN的基础知识，推荐阅读colah的文章[^4]入门。PyTorch中实现了如今最常用的三种RNN：RNN（vanilla RNN）、LSTM和GRU。此外还有对应的三种RNNCell。\n",
    "\n",
    "RNN和RNNCell层的区别在于前者一次能够处理整个序列，而后者一次只处理序列中一个时间点的数据，前者封装更完备更易于使用，后者更具灵活性。实际上RNN层的一种后端实现方式就是调用RNNCell来实现的。\n",
    "[^4]: http://colah.github.io/posts/2015-08-Understanding-LSTMs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[-0.3610, -0.1643,  0.1631],\n",
       "         [-0.0613, -0.4937, -0.1642],\n",
       "         [ 0.5080, -0.4175,  0.2502]],\n",
       "\n",
       "        [[-0.0703, -0.0393, -0.0429],\n",
       "         [ 0.2085, -0.3005, -0.2686],\n",
       "         [ 0.1482, -0.4728,  0.1425]]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.manual_seed(1000)\n",
    "# 输入：batch_size=3，序列长度都为2，序列中每个元素占4维\n",
    "input = t.randn(2, 3, 4)\n",
    "# lstm输入向量4维，隐藏元3，1层\n",
    "lstm = nn.LSTM(4, 3, 1)\n",
    "# 初始状态：1层，batch_size=3，3个隐藏元\n",
    "h0 = t.randn(1, 3, 3)\n",
    "c0 = t.randn(1, 3, 3)\n",
    "out, hn = lstm(input, (h0, c0))\n",
    "out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[-0.3610, -0.1643,  0.1631],\n",
       "         [-0.0613, -0.4937, -0.1642],\n",
       "         [ 0.5080, -0.4175,  0.2502]],\n",
       "\n",
       "        [[-0.0703, -0.0393, -0.0429],\n",
       "         [ 0.2085, -0.3005, -0.2686],\n",
       "         [ 0.1482, -0.4728,  0.1425]]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.manual_seed(1000)\n",
    "input = t.randn(2, 3, 4)\n",
    "# 一个LSTMCell对应的层数只能是一层\n",
    "lstm = nn.LSTMCell(4, 3)\n",
    "hx = t.randn(3, 3)\n",
    "cx = t.randn(3, 3)\n",
    "out = []\n",
    "for i_ in input:\n",
    "    hx, cx=lstm(i_, (hx, cx))\n",
    "    out.append(hx)\n",
    "t.stack(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "词向量在自然语言中应用十分普及，PyTorch同样提供了Embedding层。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 有4个词，每个词用5维的向量表示\n",
    "embedding = nn.Embedding(4, 5)\n",
    "# 可以用预训练好的词向量初始化embedding\n",
    "embedding.weight.data = t.arange(0,20).view(4,5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 15.,  16.,  17.,  18.,  19.],\n",
       "        [ 10.,  11.,  12.,  13.,  14.],\n",
       "        [  5.,   6.,   7.,   8.,   9.]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input = t.arange(3, 0, -1).long()\n",
    "output = embedding(input)\n",
    "output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1.4 损失函数\n",
    "在深度学习中要用到各种各样的损失函数（loss function），这些损失函数可看作是一种特殊的layer，PyTorch也将这些损失函数实现为`nn.Module`的子类。然而在实际使用中通常将这些loss function专门提取出来，和主模型互相独立。详细的loss使用请参照文档[^5]，这里以分类中最常用的交叉熵损失CrossEntropyloss为例说明。\n",
    "[^5]: http://pytorch.org/docs/nn.html#loss-functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.5944)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# batch_size=3，计算对应每个类别的分数（只有两个类别）\n",
    "score = t.randn(3, 2)\n",
    "# 三个样本分别属于1，0，1类，label必须是LongTensor\n",
    "label = t.Tensor([1, 0, 1]).long()\n",
    "\n",
    "# loss与普通的layer无差异\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "loss = criterion(score, label)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 优化器\n",
    "\n",
    "PyTorch将深度学习中常用的优化方法全部封装在`torch.optim`中，其设计十分灵活，能够很方便的扩展成自定义的优化方法。\n",
    "\n",
    "所有的优化方法都是继承基类`optim.Optimizer`，并实现了自己的优化步骤。下面就以最基本的优化方法——随机梯度下降法（SGD）举例说明。这里需重点掌握：\n",
    "\n",
    "- 优化方法的基本使用方法\n",
    "- 如何对模型的不同部分设置不同的学习率\n",
    "- 如何调整学习率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 首先定义一个LeNet网络\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.features = nn.Sequential(\n",
    "                    nn.Conv2d(3, 6, 5),\n",
    "                    nn.ReLU(),\n",
    "                    nn.MaxPool2d(2,2),\n",
    "                    nn.Conv2d(6, 16, 5),\n",
    "                    nn.ReLU(),\n",
    "                    nn.MaxPool2d(2,2)\n",
    "        )\n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Linear(16 * 5 * 5, 120),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(120, 84),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(84, 10)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.features(x)\n",
    "        x = x.view(-1, 16 * 5 * 5)\n",
    "        x = self.classifier(x)\n",
    "        return x\n",
    "\n",
    "net = Net()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from torch import  optim\n",
    "optimizer = optim.SGD(params=net.parameters(), lr=1)\n",
    "optimizer.zero_grad() # 梯度清零，等价于net.zero_grad()\n",
    "\n",
    "input = t.randn(1, 3, 32, 32)\n",
    "output = net(input)\n",
    "output.backward(output) # fake backward\n",
    "\n",
    "optimizer.step() # 执行优化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGD (\n",
       "Parameter Group 0\n",
       "    dampening: 0\n",
       "    lr: 1e-05\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       "\n",
       "Parameter Group 1\n",
       "    dampening: 0\n",
       "    lr: 0.01\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       ")"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 为不同子网络设置不同的学习率，在finetune中经常用到\n",
    "# 如果对某个参数不指定学习率，就使用最外层的默认学习率\n",
    "optimizer =optim.SGD([\n",
    "                {'params': net.features.parameters()}, # 学习率为1e-5\n",
    "                {'params': net.classifier.parameters(), 'lr': 1e-2}\n",
    "            ], lr=1e-5)\n",
    "optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGD (\n",
       "Parameter Group 0\n",
       "    dampening: 0\n",
       "    lr: 0.001\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       "\n",
       "Parameter Group 1\n",
       "    dampening: 0\n",
       "    lr: 0.01\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       ")"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 只为两个全连接层设置较大的学习率，其余层的学习率较小\n",
    "special_layers = nn.ModuleList([net.classifier[0], net.classifier[3]])\n",
    "special_layers_params = list(map(id, special_layers.parameters()))\n",
    "base_params = filter(lambda p: id(p) not in special_layers_params,\n",
    "                     net.parameters())\n",
    "\n",
    "optimizer = t.optim.SGD([\n",
    "            {'params': base_params},\n",
    "            {'params': special_layers.parameters(), 'lr': 0.01}\n",
    "        ], lr=0.001 )\n",
    "optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于如何调整学习率，主要有两种做法。一种是修改optimizer.param_groups中对应的学习率，另一种是更简单也是较为推荐的做法——新建优化器，由于optimizer十分轻量级，构建开销很小，故而可以构建新的optimizer。但是后者对于使用动量的优化器（如Adam），会丢失动量等状态信息，可能会造成损失函数的收敛出现震荡等情况。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGD (\n",
       "Parameter Group 0\n",
       "    dampening: 0\n",
       "    lr: 1e-05\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       "\n",
       "Parameter Group 1\n",
       "    dampening: 0\n",
       "    lr: 0.010000000000000002\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       ")"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 方法1: 调整学习率，新建一个optimizer\n",
    "old_lr = 0.1\n",
    "optimizer1 =optim.SGD([\n",
    "                {'params': net.features.parameters()},\n",
    "                {'params': net.classifier.parameters(), 'lr': old_lr*0.1}\n",
    "            ], lr=1e-5)\n",
    "optimizer1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGD (\n",
       "Parameter Group 0\n",
       "    dampening: 0\n",
       "    lr: 1.0000000000000002e-06\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       "\n",
       "Parameter Group 1\n",
       "    dampening: 0\n",
       "    lr: 0.0010000000000000002\n",
       "    momentum: 0\n",
       "    nesterov: False\n",
       "    weight_decay: 0\n",
       ")"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 方法2: 调整学习率, 手动decay, 保存动量\n",
    "for param_group in optimizer.param_groups:\n",
    "    param_group['lr'] *= 0.1 # 学习率为之前的0.1倍\n",
    "optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 nn.functional\n",
    "\n",
    "nn中还有一个很常用的模块：`nn.functional`，nn中的大多数layer，在`functional`中都有一个与之相对应的函数。`nn.functional`中的函数和`nn.Module`的主要区别在于，用nn.Module实现的layers是一个特殊的类，都是由`class layer(nn.Module)`定义，会自动提取可学习的参数。而`nn.functional`中的函数更像是纯函数，由`def function(input)`定义。下面举例说明functional的使用，并指出二者的不同之处。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1,  1,  1,  1],\n",
       "        [ 1,  1,  1,  1]], dtype=torch.uint8)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input = t.randn(2, 3)\n",
    "model = nn.Linear(3, 4)\n",
    "output1 = model(input)\n",
    "output2 = nn.functional.linear(input, model.weight, model.bias)\n",
    "output1 == output2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1,  1,  1],\n",
       "        [ 1,  1,  1]], dtype=torch.uint8)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = nn.functional.relu(input)\n",
    "b2 = nn.ReLU()(input)\n",
    "b == b2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此时读者可能会问，应该什么时候使用nn.Module，什么时候使用nn.functional呢？答案很简单，如果模型有可学习的参数，最好用nn.Module，否则既可以使用nn.functional也可以使用nn.Module，二者在性能上没有太大差异，具体的使用取决于个人的喜好。如激活函数（ReLU、sigmoid、tanh），池化（MaxPool）等层由于没有可学习参数，则可以使用对应的functional函数代替，而对于卷积、全连接等具有可学习参数的网络建议使用nn.Module。下面举例说明，如何在模型中搭配使用nn.Module和nn.functional。另外虽然dropout操作也没有可学习操作，但建议还是使用`nn.Dropout`而不是`nn.functional.dropout`，因为dropout在训练和测试两个阶段的行为有所差别，使用`nn.Module`对象能够通过`model.eval`操作加以区分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.nn import functional as F\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.pool(F.relu(self.conv1(x)), 2)\n",
    "        x = F.pool(F.relu(self.conv2(x)), 2)\n",
    "        x = x.view(-1, 16 * 5 * 5)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于不具备可学习参数的层（激活层、池化层等），将它们用函数代替，这样则可以不用放置在构造函数`__init__`中。对于有可学习参数的模块，也可以用functional来代替，只不过实现起来较为繁琐，需要手动定义参数parameter，如前面实现自定义的全连接层，就可将weight和bias两个参数单独拿出来，在构造函数中初始化为parameter。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyLinear(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(MyLinear, self).__init__()\n",
    "        self.weight = nn.Parameter(t.randn(3, 4))\n",
    "        self.bias = nn.Parameter(t.zeros(3))\n",
    "    def forward(self):\n",
    "        return F.linear(input, weight, bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "关于nn.functional的设计初衷，以及它和nn.Module更多的比较说明，可参看论坛的讨论和作者说明[^6]。\n",
    "[^6]: https://discuss.pytorch.org/search?q=nn.functional"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 初始化策略\n",
    "在深度学习中参数的初始化十分重要，良好的初始化能让模型更快收敛，并达到更高水平，而糟糕的初始化则可能使得模型迅速瘫痪。PyTorch中nn.Module的模块参数都采取了较为合理的初始化策略，因此一般不用我们考虑，当然我们也可以用自定义初始化去代替系统的默认初始化。而当我们在使用Parameter时，自定义初始化则尤为重要，因t.Tensor()返回的是内存中的随机数，很可能会有极大值，这在实际训练网络中会造成溢出或者梯度消失。PyTorch中`nn.init`模块就是专门为初始化而设计，如果某种初始化策略`nn.init`不提供，用户也可以自己直接初始化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.3535,  0.1427,  0.0330],\n",
       "        [ 0.3321, -0.2416, -0.0888],\n",
       "        [-0.8140,  0.2040, -0.5493],\n",
       "        [-0.3010, -0.4769, -0.0311]])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 利用nn.init初始化\n",
    "from torch.nn import init\n",
    "linear = nn.Linear(3, 4)\n",
    "\n",
    "t.manual_seed(1)\n",
    "# 等价于 linear.weight.data.normal_(0, std)\n",
    "init.xavier_normal_(linear.weight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.3535,  0.1427,  0.0330],\n",
       "        [ 0.3321, -0.2416, -0.0888],\n",
       "        [-0.8140,  0.2040, -0.5493],\n",
       "        [-0.3010, -0.4769, -0.0311]])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 直接初始化\n",
    "import math\n",
    "t.manual_seed(1)\n",
    "\n",
    "# xavier初始化的计算公式\n",
    "std = math.sqrt(2)/math.sqrt(7.)\n",
    "linear.weight.data.normal_(0,std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 对模型的所有参数进行初始化\n",
    "for name, params in net.named_parameters():\n",
    "    if name.find('linear') != -1:\n",
    "        # init linear\n",
    "        params[0] # weight\n",
    "        params[1] # bias\n",
    "    elif name.find('conv') != -1:\n",
    "        pass\n",
    "    elif name.find('norm') != -1:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5 nn.Module深入分析\n",
    "\n",
    "如果想要更深入地理解nn.Module，究其原理是很有必要的。首先来看看nn.Module基类的构造函数：\n",
    "```python\n",
    "def __init__(self):\n",
    "    self._parameters = OrderedDict()\n",
    "    self._modules = OrderedDict()\n",
    "    self._buffers = OrderedDict()\n",
    "    self._backward_hooks = OrderedDict()\n",
    "    self._forward_hooks = OrderedDict()\n",
    "    self.training = True\n",
    "```\n",
    "其中每个属性的解释如下：\n",
    "\n",
    "- `_parameters`：字典，保存用户直接设置的parameter，`self.param1 = nn.Parameter(t.randn(3, 3))`会被检测到，在字典中加入一个key为'param'，value为对应parameter的item。而self.submodule = nn.Linear(3, 4)中的parameter则不会存于此。\n",
    "- `_modules`：子module，通过`self.submodel = nn.Linear(3, 4)`指定的子module会保存于此。\n",
    "- `_buffers`：缓存。如batchnorm使用momentum机制，每次前向传播需用到上一次前向传播的结果。\n",
    "- `_backward_hooks`与`_forward_hooks`：钩子技术，用来提取中间变量，类似variable的hook。\n",
    "- `training`：BatchNorm与Dropout层在训练阶段和测试阶段中采取的策略不同，通过判断training值来决定前向传播策略。\n",
    "\n",
    "上述几个属性中，`_parameters`、`_modules`和`_buffers`这三个字典中的键值，都可以通过`self.key`方式获得，效果等价于`self._parameters['key']`.\n",
    "\n",
    "下面举例说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Net(\n",
       "  (submodel1): Linear(in_features=3, out_features=4, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        # 等价与self.register_parameter('param1' ,nn.Parameter(t.randn(3, 3)))\n",
    "        self.param1 = nn.Parameter(t.rand(3, 3))\n",
    "        self.submodel1 = nn.Linear(3, 4) \n",
    "    def forward(self, input):\n",
    "        x = self.param1.mm(input)\n",
    "        x = self.submodel1(x)\n",
    "        return x\n",
    "net = Net()\n",
    "net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('submodel1', Linear(in_features=3, out_features=4, bias=True))])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net._modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('param1', Parameter containing:\n",
       "              tensor([[ 0.3398,  0.5239,  0.7981],\n",
       "                      [ 0.7718,  0.0112,  0.8100],\n",
       "                      [ 0.6397,  0.9743,  0.8300]]))])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net._parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.3398,  0.5239,  0.7981],\n",
       "        [ 0.7718,  0.0112,  0.8100],\n",
       "        [ 0.6397,  0.9743,  0.8300]])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.param1 # 等价于net._parameters['param1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "param1 torch.Size([3, 3])\n",
      "submodel1.weight torch.Size([4, 3])\n",
      "submodel1.bias torch.Size([4])\n"
     ]
    }
   ],
   "source": [
    "for name, param in net.named_parameters():\n",
    "    print(name, param.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Net(\n",
      "  (submodel1): Linear(in_features=3, out_features=4, bias=True)\n",
      ")\n",
      "submodel1 Linear(in_features=3, out_features=4, bias=True)\n"
     ]
    }
   ],
   "source": [
    "for name, submodel in net.named_modules():\n",
    "    print(name, submodel)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('running_mean', tensor(1.00000e-02 *\n",
       "                     [ 5.1362,  7.4864])),\n",
       "             ('running_var', tensor([ 0.9116,  0.9068]))])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bn = nn.BatchNorm1d(2)\n",
    "input = t.rand(3, 2)\n",
    "output = bn(input)\n",
    "bn._buffers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nn.Module在实际使用中可能层层嵌套，一个module包含若干个子module，每一个子module又包含了更多的子module。为方便用户访问各个子module，nn.Module实现了很多方法，如函数`children`可以查看直接子module，函数`module`可以查看所有的子module（包括当前module）。与之相对应的还有函数`named_childen`和`named_modules`，其能够在返回module列表的同时返回它们的名字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0.,   2.,   0.,   0.],\n",
       "        [  8.,   0.,  12.,  14.],\n",
       "        [ 16.,   0.,   0.,  22.]])"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input = t.arange(0, 12).view(3, 4)\n",
    "model = nn.Dropout()\n",
    "# 在训练阶段，会有一半左右的数被随机置为0\n",
    "model(input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0.,   1.,   2.,   3.],\n",
       "        [  4.,   5.,   6.,   7.],\n",
       "        [  8.,   9.,  10.,  11.]])"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.training  = False\n",
    "# 在测试阶段，dropout什么都不做\n",
    "model(input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于batchnorm、dropout、instancenorm等在训练和测试阶段行为差距巨大的层，如果在测试时不将其training值设为True，则可能会有很大影响，这在实际使用中要千万注意。虽然可通过直接设置`training`属性，来将子module设为train和eval模式，但这种方式较为繁琐，因如果一个模型具有多个dropout层，就需要为每个dropout层指定training属性。更为推荐的做法是调用`model.train()`函数，它会将当前module及其子module中的所有training属性都设为True，相应的，`model.eval()`函数会把training属性都设为False。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True True\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(False, False)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(net.training, net.submodel1.training)\n",
    "net.eval()\n",
    "net.training, net.submodel1.training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('', Net(\n",
       "    (submodel1): Linear(in_features=3, out_features=4, bias=True)\n",
       "  )), ('submodel1', Linear(in_features=3, out_features=4, bias=True))]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(net.named_modules())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`register_forward_hook`与`register_backward_hook`，这两个函数的功能类似于variable函数的`register_hook`，可在module前向传播或反向传播时注册钩子。每次前向传播执行结束后会执行钩子函数（hook）。前向传播的钩子函数具有如下形式：`hook(module, input, output) -> None`，而反向传播则具有如下形式：`hook(module, grad_input, grad_output) -> Tensor or None`。钩子函数不应修改输入和输出，并且在使用后应及时删除，以避免每次都运行钩子增加运行负载。钩子函数主要用在获取某些中间结果的情景，如中间某一层的输出或某一层的梯度。这些结果本应写在forward函数中，但如果在forward函数中专门加上这些处理，可能会使处理逻辑比较复杂，这时候使用钩子技术就更合适一些。下面考虑一种场景，有一个预训练好的模型，需要提取模型的某一层（不是最后一层）的输出作为特征进行分类，但又不希望修改其原有的模型定义文件，这时就可以利用钩子函数。下面给出实现的伪代码。\n",
    "```python\n",
    "model = VGG()\n",
    "features = t.Tensor()\n",
    "def hook(module, input, output):\n",
    "    '''把这层的输出拷贝到features中'''\n",
    "    features.copy_(output.data)\n",
    "    \n",
    "handle = model.layer8.register_forward_hook(hook)\n",
    "_ = model(input)\n",
    "# 用完hook后删除\n",
    "handle.remove()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`nn.Module`对象在构造函数中的行为看起来有些怪异，如果想要真正掌握其原理，就需要看两个魔法方法`__getattr__`和`__setattr__`。在Python中有两个常用的buildin方法`getattr`和`setattr`，`getattr(obj, 'attr1')`等价于`obj.attr`，如果`getattr`函数无法找到所需属性，Python会转而调用`obj.__getattr__('attr1')`方法，即`getattr`函数无法找到的交给`__getattr__`函数处理，没有实现`__getattr__`或者`__getattr__`也无法处理的就会raise AttributeError。`setattr(obj, 'name', value)`等价于`obj.name=value`，如果obj对象实现了`__setattr__`方法，setattr会直接调用`obj.__setattr__('name', value)`，否则调用buildin方法。总结一下：\n",
    "- result  = obj.name会调用buildin函数`getattr(obj, 'name')`，如果该属性找不到，会调用`obj.__getattr__('name')`\n",
    "- obj.name = value会调用buildin函数`setattr(obj, 'name', value)`，如果obj对象实现了`__setattr__`方法，`setattr`会直接调用`obj.__setattr__('name', value')`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nn.Module实现了自定义的`__setattr__`函数，当执行`module.name=value`时，会在`__setattr__`中判断value是否为`Parameter`或`nn.Module`对象，如果是则将这些对象加到`_parameters`和`_modules`两个字典中，而如果是其它类型的对象，如`Variable`、`list`、`dict`等，则调用默认的操作，将这个值保存在`__dict__`中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('param', Parameter containing:\n",
       "              tensor([[ 1.,  1.],\n",
       "                      [ 1.,  1.]]))])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module = nn.Module()\n",
    "module.param = nn.Parameter(t.ones(2, 2))\n",
    "module._parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_modules:  OrderedDict()\n",
      "__dict__['submodules']: [Linear(in_features=2, out_features=2, bias=True), Linear(in_features=2, out_features=2, bias=True)]\n"
     ]
    }
   ],
   "source": [
    "submodule1 = nn.Linear(2, 2)\n",
    "submodule2 = nn.Linear(2, 2)\n",
    "module_list =  [submodule1, submodule2]\n",
    "# 对于list对象，调用buildin函数，保存在__dict__中\n",
    "module.submodules = module_list\n",
    "print('_modules: ', module._modules)\n",
    "print(\"__dict__['submodules']:\",module.__dict__.get('submodules'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ModuleList is instance of nn.Module:  True\n",
      "_modules:  OrderedDict([('submodules', ModuleList(\n",
      "  (0): Linear(in_features=2, out_features=2, bias=True)\n",
      "  (1): Linear(in_features=2, out_features=2, bias=True)\n",
      "))])\n",
      "__dict__['submodules']: None\n"
     ]
    }
   ],
   "source": [
    "module_list = nn.ModuleList(module_list)\n",
    "module.submodules = module_list\n",
    "print('ModuleList is instance of nn.Module: ', isinstance(module_list, nn.Module))\n",
    "print('_modules: ', module._modules)\n",
    "print(\"__dict__['submodules']:\", module.__dict__.get('submodules'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因`_modules`和`_parameters`中的item未保存在`__dict__`中，所以默认的getattr方法无法获取它，因而`nn.Module`实现了自定义的`__getattr__`方法，如果默认的`getattr`无法处理，就调用自定义的`__getattr__`方法，尝试从`_modules`、`_parameters`和`_buffers`这三个字典中获取。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "getattr(module, 'training') # 等价于module.training\n",
    "# error\n",
    "# module.__getattr__('training')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module.attr1 = 2\n",
    "getattr(module, 'attr1')\n",
    "# 报错\n",
    "# module.__getattr__('attr1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 1.,  1.],\n",
       "        [ 1.,  1.]])"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 即module.param, 会调用module.__getattr__('param')\n",
    "getattr(module, 'param')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在PyTorch中保存模型十分简单，所有的Module对象都具有state_dict()函数，返回当前Module所有的状态数据。将这些状态数据保存后，下次使用模型时即可利用`model.load_state_dict()`函数将状态加载进来。优化器（optimizer）也有类似的机制，不过一般并不需要保存优化器的运行状态。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 保存模型\n",
    "t.save(net.state_dict(), 'net.pth')\n",
    "\n",
    "# 加载已保存的模型\n",
    "net2 = Net()\n",
    "net2.load_state_dict(t.load('net.pth'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实际上还有另外一种保存方法，但因其严重依赖模型定义方式及文件路径结构等，很容易出问题，因而不建议使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type Net. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Net(\n",
       "  (submodel1): Linear(in_features=3, out_features=4)\n",
       ")"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.save(net, 'net_all.pth')\n",
    "net2 = t.load('net_all.pth')\n",
    "net2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将Module放在GPU上运行也十分简单，只需两步：\n",
    "- model = model.cuda()：将模型的所有参数转存到GPU\n",
    "- input.cuda()：将输入数据也放置到GPU上\n",
    "\n",
    "至于如何在多个GPU上并行计算，PyTorch也提供了两个函数，可实现简单高效的并行GPU计算\n",
    "- nn.parallel.data_parallel(module, inputs, device_ids=None, output_device=None, dim=0, module_kwargs=None)\n",
    "- class torch.nn.DataParallel(module, device_ids=None, output_device=None, dim=0)\n",
    "\n",
    "可见二者的参数十分相似，通过`device_ids`参数可以指定在哪些GPU上进行优化，output_device指定输出到哪个GPU上。唯一的不同就在于前者直接利用多GPU并行计算得出结果，而后者则返回一个新的module，能够自动在多GPU上进行并行加速。\n",
    "\n",
    "```\n",
    "# method 1\n",
    "new_net = nn.DataParallel(net, device_ids=[0, 1])\n",
    "output = new_net(input)\n",
    "\n",
    "# method 2\n",
    "output = nn.parallel.data_parallel(new_net, input, device_ids=[0, 1])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DataParallel并行的方式，是将输入一个batch的数据均分成多份，分别送到对应的GPU进行计算，各个GPU得到的梯度累加。与Module相关的所有数据也都会以浅复制的方式复制多份，在此需要注意，在module中属性应该是只读的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.6 nn和autograd的关系\n",
    "nn.Module利用的也是autograd技术，其主要工作是实现前向传播。在forward函数中，nn.Module对输入的tensor进行的各种操作，本质上都是用到了autograd技术。这里需要对比autograd.Function和nn.Module之间的区别：\n",
    "- autograd.Function利用了Tensor对autograd技术的扩展，为autograd实现了新的运算op，不仅要实现前向传播还要手动实现反向传播\n",
    "- nn.Module利用了autograd技术，对nn的功能进行扩展，实现了深度学习中更多的层。只需实现前向传播功能，autograd即会自动实现反向传播\n",
    "- nn.functional是一些autograd操作的集合，是经过封装的函数\n",
    "\n",
    "作为两大类扩充PyTorch接口的方法，我们在实际使用中应该如何选择呢？如果某一个操作，在autograd中尚未支持，那么只能实现Function接口对应的前向传播和反向传播。如果某些时候利用autograd接口比较复杂，则可以利用Function将多个操作聚合，实现优化，正如第三章所实现的`Sigmoid`一样，比直接利用autograd低级别的操作要快。而如果只是想在深度学习中增加某一层，使用nn.Module进行封装则更为简单高效。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.7 小试牛刀：搭建ResNet\n",
    "Kaiming He的深度残差网络（ResNet）[^7]在深度学习的发展中起到了很重要的作用，ResNet不仅一举拿下了当年CV下多个比赛项目的冠军，更重要的是这一结构解决了训练极深网络时的梯度消失问题。\n",
    "\n",
    "首先来看看ResNet的网络结构，这里选取的是ResNet的一个变种：ResNet34。ResNet的网络结构如图4-2所示，可见除了最开始的卷积池化和最后的池化全连接之外，网络中有很多结构相似的单元，这些重复单元的共同点就是有个跨层直连的shortcut。ResNet中将一个跨层直连的单元称为Residual block，其结构如图4-3所示，左边部分是普通的卷积网络结构，右边是直连，但如果输入和输出的通道数不一致，或其步长不为1，那么就需要有一个专门的单元将二者转成一致，使其可以相加。\n",
    "\n",
    "另外我们可以发现Residual block的大小也是有规律的，在最开始的pool之后有连续的几个一模一样的Residual block单元，这些单元的通道数一样，在这里我们将这几个拥有多个Residual block单元的结构称之为layer，注意和之前讲的layer区分开来，这里的layer是几个层的集合。\n",
    "\n",
    "考虑到Residual block和layer出现了多次，我们可以把它们实现为一个子Module或函数。这里我们将Residual block实现为一个子moduke，而将layer实现为一个函数。下面是实现代码，规律总结如下：\n",
    "\n",
    "- 对于模型中的重复部分，实现为子module或用函数生成相应的module`make_layer`\n",
    "- nn.Module和nn.Functional结合使用\n",
    "- 尽量使用`nn.Seqential`\n",
    "\n",
    "![图4-2: ResNet34网络结构](imgs/resnet1.png)\n",
    "![图4-3: Residual block 结构图](imgs/residual.png)\n",
    " [^7]: He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import  nn\n",
    "import torch as t\n",
    "from torch.nn import  functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ResidualBlock(nn.Module):\n",
    "    '''\n",
    "    实现子module: Residual Block\n",
    "    '''\n",
    "    def __init__(self, inchannel, outchannel, stride=1, shortcut=None):\n",
    "        super(ResidualBlock, self).__init__()\n",
    "        self.left = nn.Sequential(\n",
    "                nn.Conv2d(inchannel,outchannel,3,stride, 1,bias=False),\n",
    "                nn.BatchNorm2d(outchannel),\n",
    "                nn.ReLU(inplace=True),\n",
    "                nn.Conv2d(outchannel,outchannel,3,1,1,bias=False),\n",
    "                nn.BatchNorm2d(outchannel) )\n",
    "        self.right = shortcut\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.left(x)\n",
    "        residual = x if self.right is None else self.right(x)\n",
    "        out += residual\n",
    "        return F.relu(out)\n",
    "\n",
    "class ResNet(nn.Module):\n",
    "    '''\n",
    "    实现主module：ResNet34\n",
    "    ResNet34 包含多个layer，每个layer又包含多个residual block\n",
    "    用子module来实现residual block，用_make_layer函数来实现layer\n",
    "    '''\n",
    "    def __init__(self, num_classes=1000):\n",
    "        super(ResNet, self).__init__()\n",
    "        # 前几层图像转换\n",
    "        self.pre = nn.Sequential(\n",
    "                nn.Conv2d(3, 64, 7, 2, 3, bias=False),\n",
    "                nn.BatchNorm2d(64),\n",
    "                nn.ReLU(inplace=True),\n",
    "                nn.MaxPool2d(3, 2, 1))\n",
    "        \n",
    "        # 重复的layer，分别有3，4，6，3个residual block\n",
    "        self.layer1 = self._make_layer( 64, 64, 3)\n",
    "        self.layer2 = self._make_layer( 64, 128, 4, stride=2)\n",
    "        self.layer3 = self._make_layer( 128, 256, 6, stride=2)\n",
    "        self.layer4 = self._make_layer( 256, 512, 3, stride=2)\n",
    "\n",
    "        #分类用的全连接\n",
    "        self.fc = nn.Linear(512, num_classes)\n",
    "    \n",
    "    def _make_layer(self,  inchannel, outchannel, block_num, stride=1):\n",
    "        '''\n",
    "        构建layer,包含多个residual block\n",
    "        '''\n",
    "        shortcut = nn.Sequential(\n",
    "                nn.Conv2d(inchannel,outchannel,1,stride, bias=False),\n",
    "                nn.BatchNorm2d(outchannel))\n",
    "        \n",
    "        layers = []\n",
    "        layers.append(ResidualBlock(inchannel, outchannel, stride, shortcut))\n",
    "        \n",
    "        for i in range(1, block_num):\n",
    "            layers.append(ResidualBlock(outchannel, outchannel))\n",
    "        return nn.Sequential(*layers)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.pre(x)\n",
    "        \n",
    "        x = self.layer1(x)\n",
    "        x = self.layer2(x)\n",
    "        x = self.layer3(x)\n",
    "        x = self.layer4(x)\n",
    "\n",
    "        x = F.avg_pool2d(x, 7)\n",
    "        x = x.view(x.size(0), -1)\n",
    "        return self.fc(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ResNet()\n",
    "input  = t.randn(1, 3, 224, 224)\n",
    "o = model(input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "感兴趣的读者可以尝试实现Google的Inception网络结构或ResNet的其它变体，看看如何能够简洁明了地实现它，实现代码尽量控制在80行以内（本例去掉空行和注释总共不超过50行）。另外，与PyTorch配套的图像工具包`torchvision`已经实现了深度学习中大多数经典的模型，其中就包括ResNet34，读者可以通过下面两行代码使用：\n",
    "```python\n",
    "from torchvision import models\n",
    "model = models.resnet34()\n",
    "```\n",
    "本例中ResNet34的实现就是参考了torchvision中的实现并做了简化，感兴趣的读者可以阅读相应的源码，比较这里的实现和torchvision中实现的不同。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
